stack commented on HBASE-15638:

Ok. Have a plan and patch for Endpoints in sub-task. Has downside of 
duplicating .protos; once shaded for core and then non-shaded (for CPEPs). 
Other downsides are temporary -- IDE usage is awkward until we go the next step 
-- or minor (relatively); e.g. we do extra copy carrying CPEP request/response 
in and out of the server.

Next steps are:

 * Checkin of shaded protos; need to add build tooling so the generation and 
shading are easy-to-do out-of-bound with the build.
 * Checkin shaded protobuf source, preferably pb3
 * Checkin our small patches to protobuf to support ByteBuffers; this latter is 
probably our Ram and Anoop.

Let me make subtasks.

Should make a branch? The subtask issue for endpoints is 14.5MB in size. 
Keeping it up is painful because it touches loads of the codebase. I should ask 
out on dev list.

> Shade protobuf
> --------------
>                 Key: HBASE-15638
>                 URL: https://issues.apache.org/jira/browse/HBASE-15638
>             Project: HBase
>          Issue Type: Bug
>          Components: Protobufs
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>         Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code}<T extends com.google.protobuf.Service,R> Map<byte[],R> 
> coprocessorService(
> Class<T> service, byte[] startKey, byte[] endKey,                             
>                 org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.

This message was sent by Atlassian JIRA

Reply via email to