[
https://issues.apache.org/jira/browse/DRILL-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249084#comment-16249084
]
Paul Rogers commented on DRILL-5957:
------------------------------------
[~tdunning], good points as usual!
The end user is really the driver of version compatibility support. Drill users
have been fairly silent on this front, so perhaps version compatibility is not
necessary in this particular case. But, based on experience with other
products, folks in production often need to:
* Upgrade clients and servers on slightly different schedules, forcing the need
for older clients to talk with newer servers or visa-versa.
* If clients are installed on desktops (such as ODBC for Tableau), then the
clients may actually be remote (or on a plane or at a customer site) at the
time of server upgrades, forcing the need for older clients to connect to new
servers.
* If users have multiple Drill clusters, then they may have different upgrade
schedules, requiring that a single client be able to speak with multiple server
versions. (In the most obvious case, a user may have version X in production,
while trying out version (X+1) in a test environment prior to upgrade.)
Perhaps there are alternatives in the big data world. Maybe two versions of the
Drill client can coexist in the same Tableau or other app? (For JDBC, this
would mean that the clients must be in separate name spaces, much as SQuirreL
does, but SQLline does not do.)
Will there be a performance hit to "transcode" vectors across versions when the
version changes? Of course. The question is, is the temporary performance hit
an acceptable cost to allow a staged upgrade? Or, would the users prefer to do
an all-at-once upgrade in order to avoid the performance hit? (And, of course,
the performance hit creates a very good incentive to upgrade...)
Finally, note that the "dbody" portion of of a Drill message exists outside of
the Protobuf structure. A Drill message has four parts:
* Message ID
* P-body (Protobuf body) length
* P-body (Serialized Protobuf content)
* D-body (data body) length
* D-body (serialized value vectors)
For this reason, Protobuf formats don't help us with vector serialization. Note
that vector data may be GB in size, so sending two copies is a worse
performance impact than transcoding...
All this said, if staged upgrades and version compatibility is not a concern
for Drill users at present, then there is no barrier to upgrading our vector
formats; we just require new clients be used with the new Drill version. This
would, of course, be the simplest solution by far.
> Wire protocol versioning, version negotiation
> ---------------------------------------------
>
> Key: DRILL-5957
> URL: https://issues.apache.org/jira/browse/DRILL-5957
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
>
> Drill has very limited support for evolving its wire protocol. As Drill
> becomes more widely deployed, this limitation will constrain the project's
> ability to rapidly evolve the wire protocol based on user experience to
> improve simplicitly, performance or minimize resource use.
> Proposed is a standard mechanism to version the API and negotiate the API
> version between client and server at connect time. The focus here is between
> Drill clients (JDBC, ODBC) and the Drill server. The same mechanism can also
> be used between servers to support rolling upgrades.
> This proposal is an outline; it is not a detailed design. The purpose here is
> to drive understanding of the problem. Once we have that, we can focus on the
> implementation details.
> h4. Problem Statement
> The problem we wish to address here concerns both the _syntax_ and
> _semantics_ of API messages. Syntax concerns:
> * The set of messages and their sequence
> * The format of bytes on the wire
> * The format of message packets
> Semantics concerns:
> * The meaning of each field.
> * The layout of non-message data (vectors, in Drill.)
> We wish to introduce a system whereby both syntax and semantics can be
> evolved in a controlled, known manner such that:
> * A client of version x can connect to, and interoperate with, a server in a
> range of versions (x-y, x+z) for some values of y and z.
> For example, version x of the Drill client is deployed in the field. It must
> connect to the oldest Drill cluster available to that client. (That is it
> must connect to servers up to y versions old.) During an upgrade, the server
> may be upgraded before the client. Thus, the client must also work with
> servers up to z versions newer than the client.
> If we wish to tackle rolling upgrades, then y and z can both be 1 for
> server-to-server APIs. A version x server will talk with (x-1) servers when
> the cluster upgrades to x, and will talk to (x+1) servers when the cluster is
> upgraded to version (x+1).
> h4. Current State
> Drill currently provides some ad-hoc version compatibility:
> * Slow change. Drill's APIs have not changed much since Drill 1.0, thereby
> avoiding the issue.
> * Protobuf support. Drill uses Protobuf for message bodies, leveraging that
> format's ability to absorb the additional or deprecation of individual fields.
> * API version number. The API holds a version number, though the code to use
> it is rather ad-hoc.
> The above has allowed clever coding to handle some version changes, but each
> is a one-off, ad-hoc collision. The recent security work is an example that,
> with enough effort, ad-hoc solutions can be found.
> The above cannot handle:
> * Change in the message order
> * Change in the "pbody/dbody" structure of each message.
> * Change in the structure of serialized value vectors.
> As a result, the current structure prevents any change to Drill's core
> mechanism, value vectors, as there is no way or clients and servers to
> negotiate the vector wire format. For example, Drill cannot adopt Arrow
> because a pre-Arrow client would not understand "dbody" message parts encoded
> in Arrow format and visa-versa.
> h4. API Version
> The core of the proposal is to introduce an API version. This is a simple
> integer which is incremented each time that a breaking change is made to the
> API. (If the change can be absorbed by the Protobuf mechanism, then it is not
> a breaking change.) Note that the API version *is not* the same as the
> product version. Two different Drill versions may have the same API version
> if nothing changed in the API.
> h4. Version Negotiation
> Given a set of well-defined protocol versions, we can next define the version
> negotiation protocol between client and server:
> * The client connects and sends a "hello" message that identifies the range
> of API versions that it supports, with the newest version being the version
> of the client itself.
> * The server receives the message and computes the version of the session as
> the newest client version the the server supports.
> * The server returns this version to the client which switches to the
> selected API version. (The server returns an error, and disconnects, if there
> is no common version.)
> * The server and client use only messages valid for the given API version.
> This may mean converting data from one representation to another.
> The above is pretty standard.
> h4. Backward Compatibility Implementation
> Consider a server that must work with its own version (version c) and, say,
> two older versions (a and b).
> In most cases, changes across versions are minor. Perhaps version b
> introduced a better error reporting format (akin to SQLWARN and SQLERROR
> codes). Version c may have change the layout of a particular vector (or
> introduce a new vector type.) How does the server handle these when talking
> with older clients?
> If a version c server talks to a version a client, then it must retain the
> old version a way of reporting errors. If the same version talks to a version
> b or c server, it uses the new form. This simply means that, at the point
> that an error response is sent to the client, the server selects either the a
> version of that message or the b version.
> Since the a version previously existed, and the b version was added, it is
> straightforward (if tedious) or the developer to retain both versions and use
> them depending on the client version.
> Now, consider version c that introduced a new vector format. Here, the server
> must transcode the data to the a version for clients of version a or b.
> Again, since the a version existed, and the c version was added, the
> developer is aware of both. The transcoding step is new, and does introduce a
> performance hit. But, it does allow the newer format to be rewritten in the
> older wire format.
> For example, perhaps version c modified offset vectors to encode the end
> position of each row, rather than the start position (with the start position
> of the first row implied to be at 0.) This causes the offset vector to shrink
> by one position. Because of power-of-two rounding, this may halve the memory
> required. If the server talks to a version a or b client, then it simply
> transcodes the vector by shifting it upward and inserting the required 0
> value at the zero position.
> The same logic applies to the client as well, when receiving data. A version
> c client, when working with a version a server, must transcode the old offset
> vector format to the c version.
> h4. Bootstrap
> The above is fine for new Drill versions. But, how would we get started? This
> is the bootstrap problem: we can't change to the new version until both
> clients and servers understand the new version. But, since Drill does not
> currently support versioning, we can't introduce versioning until Drill
> understands versioning: a catch-22 situation.
> We can observe that, initially, we are mostly concerned with client/server
> communication: old clients will exist in the field even after the upgrade to
> the version-aware server. We also observe that the client initiates the
> conversation.
> One solution is to leverage the existing client "hello" message. Suppose we
> introduce the above versioning protocol in API version 10 (say). When a 9 or
> older version client connects, it will send the existing Protobuf "hello"
> message. The version 10 server can use that to detect the older client, and
> immediately switch to the old (unversioned) protocol.
> Version 10 clients introduce a new V2 "hello" message that contains the
> schema negotiation information. If the server receives this, then the server
> knows it is working with a version 10 (or later) client, and will use that
> protocol instead, complete with version negotiation and adaptation as
> described above.
> h4. Alternatives
> One solution is to require uses to upgrade their drivers at the same moment
> that they upgrade their servers. This is often impractical, however, and is
> not possible if a single driver must work with multiple Drill servers
> upgraded at different times.
> Suppose that we decide not to introduce versioning at the API level, but
> instead soldier on with what we have. We will be able to:
> * Carefully introduce new messages and new Protobuf fields.
> We will *not* be able to:
> * Change the wire format for any non-Protobuf data, especially value vectors.
> The result is that we will be frozen at the Drill 1.0 version of the value
> vector format (and so we must hope that that original format was prescient
> about our future needs.) We cannot improve vector efficiency, nor can we
> switch to the Arrow format.
> Unfortunately, our competitors will continue to move forward with their
> formats, placing Drill at a competitive disadvantage. So, in order to remain
> competitive, we have little alternative.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
