[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535165#comment-15535165
]
stack commented on HBASE-15638:
-------------------------------
Update:
* HBASE-16264 subtask is good to go; does shading of protobuf but makes it so
CPEPs keep working
* HBASE-16567, upgrade to 3.1 seems to work. Patch depends on HBASE-16264
being in first.
Currently working on the checking in shaded protobuf and shaded generated
source. Trying to elaborate on our current generate-protos step so it does
generation, shading, and copy-local of the shaded protobuf files. Idea is we
check this all in and thereafter the build will use the generated sources and
stuff like IDEs will just work. If we need to patch the protobufs, this will be
done as part of this -Pcompile-protobuf out-of-band step.
Have my little jenkins build and branch going on to help work through this
stuff:
https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/
https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=log;h=refs/heads/HBASE-16264
> Shade protobuf
> --------------
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch,
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch,
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch,
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch,
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch,
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch,
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch,
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs
> expect all buffers to be on-heap byte arrays. It does not have facility for
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact
> frustrates the off-heaping-of-the-write-path project as
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To
> ensure we pick up the patched protobuf always, we need to relocate/shade our
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints --
> which use protobuf Service to describe new API -- a blind relocation/shading
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP)
> in particular. For example, in the Table Interface, to invoke a method on a
> registered CPEP, we have:
> {code}<T extends com.google.protobuf.Service,R> Map<byte[],R>
> coprocessorService(
> Class<T> service, byte[] startKey, byte[] endKey,
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R>
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while
> preserving our API as is so CPEPs continue to work on the new hbase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
