Paul Rogers created DRILL-5957:
----------------------------------
Summary: Wire protocol versioning, version negotiation
Key: DRILL-5957
URL: https://issues.apache.org/jira/browse/DRILL-5957
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.11.0
Reporter: Paul Rogers
Drill has very limited support for evolving its wire protocol. As Drill becomes
more widely deployed, this limitation will constrain the project's ability to
rapidly evolve the wire protocol based on user experience to improve
simplicitly, performance or minimize resource use.
Proposed is a standard mechanism to version the API and negotiate the API
version between client and server at connect time. The focus here is between
Drill clients (JDBC, ODBC) and the Drill server. The same mechanism can also be
used between servers to support rolling upgrades.
This proposal is an outline; it is not a detailed design. The purpose here is
to drive understanding of the problem. Once we have that, we can focus on the
implementation details.
h4. Problem Statement
The problem we wish to address here concerns both the _syntax_ and _semantics_
of API messages. Syntax concerns:
* The set of messages and their sequence
* The format of bytes on the wire
* The format of message packets
Semantics concerns:
* The meaning of each field.
* The layout of non-message data (vectors, in Drill.)
We wish to introduce a system whereby both syntax and semantics can be evolved
in a controlled, known manner such that:
* A client of version x can connect to, and interoperate with, a server in a
range of versions (x-y, x+z) for some values of y and z.
For example, version x of the Drill client is deployed in the field. It must
connect to the oldest Drill cluster available to that client. (That is it must
connect to servers up to y versions old.) During an upgrade, the server may be
upgraded before the client. Thus, the client must also work with servers up to
z versions newer than the client.
If we wish to tackle rolling upgrades, then y and z can both be 1 for
server-to-server APIs. A version x server will talk with (x-1) servers when the
cluster upgrades to x, and will talk to (x+1) servers when the cluster is
upgraded to version (x+1).
h4. Current State
Drill currently provides some ad-hoc version compatibility:
* Slow change. Drill's APIs have not changed much since Drill 1.0, thereby
avoiding the issue.
* Protobuf support. Drill uses Protobuf for message bodies, leveraging that
format's ability to absorb the additional or deprecation of individual fields.
* API version number. The API holds a version number, though the code to use it
is rather ad-hoc.
The above has allowed clever coding to handle some version changes, but each is
a one-off, ad-hoc collision. The recent security work is an example that, with
enough effort, ad-hoc solutions can be found.
The above cannot handle:
* Change in the message order
* Change in the "pbody/dbody" structure of each message.
* Change in the structure of serialized value vectors.
As a result, the current structure prevents any change to Drill's core
mechanism, value vectors, as there is no way or clients and servers to
negotiate the vector wire format. For example, Drill cannot adopt Arrow because
a pre-Arrow client would not understand "dbody" message parts encoded in Arrow
format and visa-versa.
h4. API Version
The core of the proposal is to introduce an API version. This is a simple
integer which is incremented each time that a breaking change is made to the
API. (If the change can be absorbed by the Protobuf mechanism, then it is not a
breaking change.) Note that the API version *is not* the same as the product
version. Two different Drill versions may have the same API version if nothing
changed in the API.
h4. Version Negotiation
Given a set of well-defined protocol versions, we can next define the version
negotiation protocol between client and server:
* The client connects and sends a "hello" message that identifies the range of
API versions that it supports, with the newest version being the version of the
client itself.
* The server receives the message and computes the version of the session as
the newest client version the the server supports.
* The server returns this version to the client which switches to the selected
API version. (The server returns an error, and disconnects, if there is no
common version.)
* The server and client use only messages valid for the given API version. This
may mean converting data from one representation to another.
The above is pretty standard.
h4. Backward Compatibility Implementation
Consider a server that must work with its own version (version c) and, say, two
older versions (a and b).
In most cases, changes across versions are minor. Perhaps version b introduced
a better error reporting format (akin to SQLWARN and SQLERROR codes). Version c
may have change the layout of a particular vector (or introduce a new vector
type.) How does the server handle these when talking with older clients?
If a version c server talks to a version a client, then it must retain the old
version a way of reporting errors. If the same version talks to a version b or
c server, it uses the new form. This simply means that, at the point that an
error response is sent to the client, the server selects either the a version
of that message or the b version.
Since the a version previously existed, and the b version was added, it is
straightforward (if tedious) or the developer to retain both versions and use
them depending on the client version.
Now, consider version c that introduced a new vector format. Here, the server
must transcode the data to the a version for clients of version a or b. Again,
since the a version existed, and the c version was added, the developer is
aware of both. The transcoding step is new, and does introduce a performance
hit. But, it does allow the newer format to be rewritten in the older wire
format.
For example, perhaps version c modified offset vectors to encode the end
position of each row, rather than the start position (with the start position
of the first row implied to be at 0.) This causes the offset vector to shrink
by one position. Because of power-of-two rounding, this may halve the memory
required. If the server talks to a version a or b client, then it simply
transcodes the vector by shifting it upward and inserting the required 0 value
at the zero position.
The same logic applies to the client as well, when receiving data. A version c
client, when working with a version a server, must transcode the old offset
vector format to the c version.
h4. Bootstrap
The above is fine for new Drill versions. But, how would we get started? This
is the bootstrap problem: we can't change to the new version until both clients
and servers understand the new version. But, since Drill does not currently
support versioning, we can't introduce versioning until Drill understands
versioning: a catch-22 situation.
We can observe that, initially, we are mostly concerned with client/server
communication: old clients will exist in the field even after the upgrade to
the version-aware server. We also observe that the client initiates the
conversation.
One solution is to leverage the existing client "hello" message. Suppose we
introduce the above versioning protocol in API version 10 (say). When a 9 or
older version client connects, it will send the existing Protobuf "hello"
message. The version 10 server can use that to detect the older client, and
immediately switch to the old (unversioned) protocol.
Version 10 clients introduce a new V2 "hello" message that contains the schema
negotiation information. If the server receives this, then the server knows it
is working with a version 10 (or later) client, and will use that protocol
instead, complete with version negotiation and adaptation as described above.
h4. Alternatives
Suppose that we decide not to introduce versioning at the API level, but
instead soldier on with what we have. We will be able to:
* Carefully introduce new messages and new Protobuf fields.
We will *not* be able to:
* Change the wire format for any non-Protobuf data, especially value vectors.
The result is that we will be frozen at the Drill 1.0 version of the value
vector format (and so we must hope that that original format was prescient
about our future needs.) We cannot improve vector efficiency, nor can we switch
to the Arrow format.
Unfortunately, our competitors will continue to move forward with their
formats, placing Drill at a competitive disadvantage. So, in order to remain
competitive, we have little alternative.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
