[
https://issues.apache.org/jira/browse/HDFS-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
junwen yang updated HDFS-15700:
-------------------------------
Description:
Regarding the issue -HDFS-9788- caused by the incompatibility of protobuf
message, we have created a static checker which keeps track of the proto file
change, and detect potential incompatibility:
# Add/delete required field, which is the case reported in HBASE-25238.
# The tag number of a field has been changed, as described in HDFS-9788. Also,
the [protobuf guidelines
|https://developers.google.com/protocol-buffers/docs/proto]suggests _each field
in the message definition has a *unique number*. These numbers are used to
identify your fields in the [message binary
format|https://developers.google.com/protocol-buffers/docs/encoding], and
should not be changed once your message type is in use_.
# A required field has been changed to optional, or an optional field has
been changed to required. According to the guidelines in [protobuf official
website|https://developers.google.com/protocol-buffers/docs/proto], _*Required
Is Forever* You should be very careful about marking fields as {{required}}. If
at some point you wish to stop writing or sending a required field, it will be
problematic to change the field to an optional field - old readers will
consider messages without this field to be incomplete and may reject or drop
them unintentionally. You should consider writing application-specific custom
validation routines for your buffers instead._
We have applied our checker on the frequently maintained HDFS versions:
rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0,
rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0,
rel/release-3.3.0, and we found 6 potential problems as attached. The checker
is also able to detect the issue describe in -HDFS-9788.-
The results reported by our checker got confirmed by developers of HBASE and
IMPALA, and our checker is requested by HBASE, which can be found at
HBASE-25340.
was:
Regarding the issue -HDFS-9788- caused by the incompatibility of protobuf
message, we have created a static checker which keeps track of the proto file
change, and detect potential incompatibility:
# Add/delete required field, which is the case reported in HBASE-25238.
# The tag number of a field has been changed, as described in HDFS-9788. Also,
the [protobuf guidelines
|https://developers.google.com/protocol-buffers/docs/proto]suggests _each field
in the message definition has a *unique number*. These numbers are used to
identify your fields in the [message binary
format|https://developers.google.com/protocol-buffers/docs/encoding], and
should not be changed once your message type is in use_.
# A required field has been changed to optional, or an optional field has
been changed to required. According to the guidelines in [protobuf official
website|https://developers.google.com/protocol-buffers/docs/proto], _*Required
Is Forever* You should be very careful about marking fields as {{required}}. If
at some point you wish to stop writing or sending a required field, it will be
problematic to change the field to an optional field - old readers will
consider messages without this field to be incomplete and may reject or drop
them unintentionally. You should consider writing application-specific custom
validation routines for your buffers instead._
We have applied our checker on the frequently maintained HDFS versions:
rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0,
rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0,
rel/release-3.3.0, and we found 6 potential problems as attached. The checker
is also able to detect the issue describe in -HDFS-9788.-
The results reported by our checker got confirmation from developers in HBASE,
which can be found here HBASE-25340.
> Protobuf Mesage Incompatibility Detector
> ----------------------------------------
>
> Key: HDFS-15700
> URL: https://issues.apache.org/jira/browse/HDFS-15700
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: junwen yang
> Priority: Major
> Attachments: hdfs_proto_incompatibility.txt
>
>
> Regarding the issue -HDFS-9788- caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is the case reported in HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HDFS versions:
> rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0,
> rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0,
> rel/release-3.3.0, and we found 6 potential problems as attached. The checker
> is also able to detect the issue describe in -HDFS-9788.-
> The results reported by our checker got confirmed by developers of HBASE and
> IMPALA, and our checker is requested by HBASE, which can be found at
> HBASE-25340.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
