[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833227#comment-13833227 ] Andrew Purtell commented on HBASE-9681: --- Where is this issue [~ram_krish] or [~anoop.hbase]? Planning on branching 0.98 Sunday 12/1. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812803#comment-13812803 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- bq.So it will be like the server will always NOT write back the cell tags? Currently it is based on the mapping that we provide and no context is available with the client. For export tool we may have to see how to over come the problem. When we have any type of negotiation 0.96 clients talking to 0.98 servers expecting them to act like 0.96 servers would be another tricky part because the we cannot use a new codec here as the 0.96 client is not aware of it. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812814#comment-13812814 ] Anoop Sam John commented on HBASE-9681: --- So here basically using a different Encoder at server side, we will or will not serialize the tags back to the client. In case of scan where we use Codec based serialization of Result cells in the PayloadCarryingRpcController's payload, this different Codec will work. But what about pure PB based request, response? In this case the cells at server side will be converted to PB cells and returned. In the current code this conversion is not considering cell tags so we are not serializing back the tags as of now. For get request, there will be PB based return. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812602#comment-13812602 ] Anoop Sam John commented on HBASE-9681: --- Ram So it will be like the server will always NOT write back the cell tags? Or that also based on some context information? Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790081#comment-13790081 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- When codec is getting used say KeyValueCodec we would be using the codec in in Puts also. Means the server would use the codec to decode the KVs and add it to the RS. The same codec will be used by the server while the client issues a scan or read operation. This means that the codec is uniform in the client and the server. For example, take the case of tags, the client should be able to add the tags through the codec while doing Puts and the server should decode the tags. Where as while doing the scan the server should suppress the tags using a different codec and the client should be able to decode KVs using that codec. Using one codec would be applicable for pure scan cases but when we have read and writes then both client and server should be aware of its own local and the remote codec on the other side. Either we could have a mapping in the server side for the codec specified on the client side or we should have specific api in the encoder and decoders like getEncoderForClient, getEncoderForServer. Anyway this mapping ( we can say as the intelligence added to server to decide the codec) is always context specific. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[
https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790099#comment-13790099
]
Elliott Clark commented on HBASE-9681:
--
That sounds more like an in and an out codec. Why not just add each way.
eg.
{code}
// This is sent on connection setup after the connection preamble is sent.
message ConnectionHeader {
optional UserInformation user_info = 1;
optional string service_name = 2;
// Cell block codec we will use sending over optional cell blocks. Server
throws exception
// if cannot deal. Null means no codec'ing going on so we are pb all the
time (SLOW!!!)
optional string cell_block_codec_class = 3;
// This one is already there.
// For 0.98 this is set to a class that's only present in 0.98+
// Something like org.apache.hadoop.hbase.codecs.SpecifyEachDirectionCodec
// Obviously the name can use some work.
optional string cell_block_compressor_class = 4;
// This will be used on all key values being sent from this client to the
sever
optional string cell_block_compressor_from_client_class = 5;
// This will be used on all all key values being sent to the client from the
server.
optional string cell_block_compressor_to_client_class = 6;
}
{code}
Older servers will throw an exception if they don't know the full duplex codec.
So the client will fall back to a 0.96 codec.
Servers after 0.98 will throw an exception if they can't accept the codec's
specified for each direction (preferably an info carrying exception with a list
of codecs that are accepted). If it's successful then we just use each the
negotiated codec.
With this there's no extra round trips on the default case (server and client
both support the same codecs). On the worst case there's one extra round trip.
Basic codec negotiation
---
Key: HBASE-9681
URL: https://issues.apache.org/jira/browse/HBASE-9681
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Basic codec negotiation:
There should be a default codec used for cell encoding over the RPC
connection. This should be configurable in the site file.
The client can optionally send a message, a manufactured call that would
otherwise be invalid in some way, to the server asking for a list of
supported cell codecs. An older server should simply send back an error
because the request is invalid except to servers supporting this feature. A
server supporting this feature should send back the requested information or
an error indication if something went wrong.
The client can optionally send a message, a manufactured call that would
otherwise be invalid in some way, to the server asking for it to use a given
codec for all further communication. Otherwise the server will continue to
use the default codec. The server will send back a call response
acknowledging the change or an error indication if the request cannot be
honored.
Server configuration should support mappings from one codec type to another.
We need to handle the case where the server has a codec available that
extends the requested type but overrides some behavior in the base class, and
this is what should be used in lieu of the base type. It must also be
possible to choose an alternate default codec which stands in for the default
codec, is compatible with client expectations, but changes the server side
behavior as needed in the absence of negotiation.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790238#comment-13790238 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- Just to clarify bq.Something like org.apache.hadoop.hbase.codecs.SpecifyEachDirectionCodec This will have the two codecs that would be needed by the client and server? Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790617#comment-13790617 ] Elliott Clark commented on HBASE-9681: -- bq.This will have the two codecs that would be needed by the client and server? I was thinking that would be the sentinel that the following two classes are used. However I'm still a little slow on picking up when you'd need the client to specify more than one codec. It seems like in your example having the client specify one codec and having the codec have some idea of where it's running would make this easier. h2.Client sending to Server # Client calls getCodec. # Client get Codec1 # Tells that codec that it's on the client # Client then calls getEncoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getEncoder(java.io.OutputStream) #* This call returns Codec1.ClientEncoder # send the kv's to the codec's encoder who encodes the key values placing the tags at the end of the byte stream # Client sends that encoded byte stream to the server after specifying the codec on the header. # Server then looks up the same codec codec by calling getCodec # Server gets Codec1 # Server then tells the codec that it's on the server # Server gets the decoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getDecoder(java.io.InputStream) #* This call returns Codec1.ServerDecoder # Server feeds the decoder the bytes who preforms the exact inverse of what was done by the client's encoder. h2.Server sending to Client # Server then looks up the codec specified in the connection header # Server gets Codec1 # Server then tells the codec that it's on the server # Server gets the encoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getDecoder(java.io.InputStream) #* This call returns Codec1.ServerEncoder # Server feeds the encoder the cells. It stips any system tags and then encodes the cells by moving the tags to the back end (or whatever is needed) # Server sends these bytes # Client then gets the Call result with the encoded bytes. # Notices they are encoded so callse getCodec # The client gets Codec1 # The client then informs Codec1 that it's on the client # The client gets the decoder #* This call returns Codec1.ClientDecoder # This decoder knows nothing about system tags so it only preforms the inverse of moving the tags to the end of the byte stream. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13791203#comment-13791203 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- Correct.. Agree. So here the mapping is we have a codec say the ClientSideCodec and the ServerSideCodec for the same type of base codec. The encoder and decoder will be called based on the process that calls it.. (either the client or the server). We have considered this way also. Let me see what changes would be needed for getting this. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13791222#comment-13791222 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- The impl that I have does the same but in a bit different way. So I would add a configuration say a mapping between two codecs. Say CodecA is related with CodecB. Client would say it would be using CodecA on the header and server would pick up CodecB from the above mapping configuration(if found). The CodecA.getEncoder and CodecB.getDecoder would work in sync (for client to server communication). The CodecB.getEncoder and CodecA.getDecoder() would work in sycn (for server to client communication). Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13791226#comment-13791226 ] Elliott Clark commented on HBASE-9681: -- Yeah that can work too(I'm not sure which would be cleaner) as long as anything that uses the mapping is a new codec so there's no possibility of older client's being confused. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1379#comment-1379 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- Regarding the mapping, how should the server decide on the mapping? Should it be from a configuration? If it is from a configuration then it is not generic because this may be context based. One reason why the mapping is important because in some cases the client may ask for a codec but it may not be right for the server to send out all the information say for example some information stored in the tags. So in that case the server should apply the context based knowledge on what codec it should use. This is where the mapping comes into picture. In an internal discussion, there is a chance that for some cases like Export the server still needs to send out information in tags too. So in that case the mapping should deal this in a different way. So overall this mapping and usage of the codecs on the server side is more context based. When we have sytem tags in future the above argument applies (generally here it is better to suppress the tags from the clients). So the connection negotiation part is a protocol that we can decide but the mapping is one which is not very generic. Can try to make this mapping a pluggable one? Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790050#comment-13790050 ] stack commented on HBASE-9681: -- Give me example of a mapping mighty [~ram_krish] so I can follow along better. Rather than one mapping for clients and another for export, would it better to have this switch be done on what permissions the client has? Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788255#comment-13788255 ] Andrew Purtell commented on HBASE-9681: --- So to modify the OP with Elliot's suggestion: The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs, and also optionally a preference ordered list of the codecs it wants. An older server should simply send back an error because the request is invalid except to servers supporting this feature. If the preference ordered list was provided, the server should enable the most preferred codec available for all further communication on the connection and send back a call response acknowledging the change and indicating which codec is in use, or an error indication if something went wrong. If a preference ordered list was not provided, or if no codec on the client's list was available, the server should send back a list of available codecs. The client can optionally send another message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use one of the codecs from the server supplied list for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Suggestions for error handling: - Once codec negotiation is initiated by the client, if the client and server can't agree on a codec then the connection setup should fail with a big fat ERROR in the logs on the server and client sides both. - We should not do anything about incompatible default codecs if codec negotiation is disabled (either due to older servers, or older clients, or a configuration toggle). Connection setup should proceed, but there will be failures later. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788309#comment-13788309 ] stack commented on HBASE-9681: -- Is this over-engineering? What if we have it that server never drops support for any one codec (at least, within major versions). Then there is no need of negotiation? Client just says what they want? Server fails connection if can't give client want it wants. I'm guessing client doesn't want 'fallback'. If it does, it can do it itself by catching the UnsupportedCodecException and then go again w/ a different codec. See 'J.4.3.1. CellBlock Codecs' in http://hbase.apache.org/book.html#d0e14417 Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788311#comment-13788311 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- The most important thing would be how we change the default behaviour of the base class in the client and in the server. We need a mapping for this and this mapping should be configurable. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788339#comment-13788339 ] Andrew Purtell commented on HBASE-9681: --- bq. [Stack] Is this over-engineering? I wanted to put this idea out there and make a concrete proposal rather than a two line JIRA. We can wait on most of this until there are mixed major version deployments or when we have different codec flavors to choose from. Don't you think we could use something like this for a 1.0 for future-proofing? bq. [Ram] The most important thing would be how we change the default behaviour of the base class in the client and in the server. We need a mapping for this and this mapping should be configurable. What Ram said. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788343#comment-13788343 ] Andrew Purtell commented on HBASE-9681: --- On the over-engineering comment, if the proposal is ugly, that's one thing, but implementing coprocessors just to put in security was arguably over-engineering at the time too. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788352#comment-13788352 ] Sergey Shelukhin commented on HBASE-9681: - Small clarification - why does there have to be an invalid message? If it's a separate method/message, when the new client attempts to send it to an old server, it will get an easily recognizable exception back, right? (Thrift has exception for no-such-method, at least; don't recall for PB) Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788357#comment-13788357 ] Andrew Purtell commented on HBASE-9681: --- bq, Small clarification - why does there have to be an invalid message? If it's a separate method/message, when the new client attempts to send it to an old server, it will get an easily recognizable exception back, right? For the old server it's an invalid message. Ok, there I agree that's over specified, I had one thing in mind but it's not the only way. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788384#comment-13788384 ] stack commented on HBASE-9681: -- bq. Don't you think we could use something like this for a 1.0 for future-proofing? Sure. Was just trying to save having to do fancy-dancing when may not be necessary (just-yet) Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788457#comment-13788457 ] Andrew Purtell commented on HBASE-9681: --- bq. Was just trying to save having to do fancy-dancing when may not be necessary (just-yet) I thought we could invest a little extra for the future while working in this area, but if I'm the only one who thinks this particular proposal is worthy, then we can just do the part Ram mentioned and address what actually needs doing right now. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784971#comment-13784971 ] Andrew Purtell commented on HBASE-9681: --- bq. And the header of which can be forcefully made negative and the client/server understands the new response/request if the header size is negative? Sounds like a nasty hack Ram. So a 0.96 client won't barf on this new request/response message you propose? How is that a different approach than my OP on this issue? Perhaps you can put up a design doc explaining. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784982#comment-13784982 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- Sure.. bq.Sounds like a nasty hack Ram. I have tried removing this way of getting the reponse back and forth. Will post a design doc sooner. I am working on a cleaner way of doing rather than making the header negative. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785261#comment-13785261 ] Elliott Clark commented on HBASE-9681: -- bq.The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. That sounds like a lot of round trips. Can we have the client send a preference ordered list of codec it wants. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785856#comment-13785856 ] Andrew Purtell commented on HBASE-9681: --- bq. That sounds like a lot of round trips. Can we have the client send a preference ordered list of codec it wants. +1 Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784821#comment-13784821 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- bq.We can do this with configuration or decorators. Either we add a new site configuration option where a comma separated list of codec classes Currently using this as configuration. We can use reflections to get the Codec related class paths too. I can later work on that part. bq. I suppose it could be done with new optional fields in the connection header and response instead but this would need to support: Yes, adding new fields should work. Working on it. I would post a WIP patch on this. I feel we may need some new request/response too to make this 2 way communication. And the header of which can be forcefully made negative and the client/server understands the new response/request if the header size is negative? Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783142#comment-13783142 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- Currently we have a connectionheader, request header and request response on the RPC communication side. The connection header is the place where the codec and compression is determined i.e once per connection. Once the connection header is processed the remaining normal Request headers are processed. So as to introduce the basic negotiaion protocol 1) Need to introduce a connection response that on getting the option from the client to return the list of codecs. So on what basis should the server return them? Check it from the classpath? 2)This response to be interpreted by the client and select one codec from the list and send that codec as a new response from the client to the server. 3) To this response, the server tries to use the codec given by the client and again returns a success/failure msg if it was able to use the specified codec. 4)The mapping of individual codecs and the extended functionality of codecs that needs specific behaviour on the server side am yet to work on that. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783612#comment-13783612 ] Andrew Purtell commented on HBASE-9681: --- bq. The connection header is the place where the codec and compression is determined i.e once per connection. My thinking is this is where the initial default codec for the connection would established like it is done now for 0.96. Then an additional round of negotiation would optionally happen, hence the idea for using 'invalid' calls after the initial connection header processing. I suppose it could be done with new optional fields in the connection header and response instead but this would need to support: - 0.96 clients talking to 0.98 servers expecting them to act like 0.96 servers - 0.98 clients talking to 0.96 servers - 0.98 clients talking to 0.98 servers, with codec negotiation, and server side substitution bq. Need to introduce a connection response that on getting the option from the client to return the list of codecs. So on what basis should the server return them? Check it from the classpath? We can do this with configuration or decorators. Either we add a new site configuration option where a comma separated list of codec classes could go, or we create a new decorator type that marks codecs and scan the classpath for the decorator at server startup and add the discovered classes to a list. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781282#comment-13781282 ] ramkrishna.s.vasudevan commented on HBASE-9681: --- Correct we need something like this. Where the client is able to tell what it would send and what the server should use. The server should respond with what server will be using from the mapping. Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: Andrew Purtell Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1#6144)
