[
https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15464927#comment-15464927
]
stack commented on HBASE-16264:
-------------------------------
I updated the attached doc. with where I am at on Approach #3. I repeat status
below. I've uncovered some new kinks while on this tack; i.e. some CPEPs are
core and Query and Mutation make use of CPEP AccessControl protos. Next up
(Approach #4) is a disruptive renaming of proto files used internally to remove
our clash on proto names.
h2. Tactic #3
Same as Tactic #1 only move the new hbase-endpoint module with all CPEPs in it
AFTER hbase-server (and hbase-client) so all CPEPs bundled in with hbase are in
the new hbase-endpoint module. CPEPs will then load their versions of
non-shaded ClientProtos, etc., rather than the systems' shaded versions. Should
work since CPEPs have own classloader so if issues, we can tinker (Root issue
is that there are two versions of proto files; versions where protobuf is
shaded used internally everywhere, and then the versions of protos that are not
shaded so CPEPs can keep working -- we can’t change Class naming/layout if we
want CPEPs to keep working).
This approach #3 fails for a reason similar to why approach #1 fails (only it
is the inverse); the unit tests in hbase-endpoint are picking up the local
non-shaded versions of ClientProtos when Master/RegionServer are starting up
the minihbasecluster. Tests are failing to run with NoSuchMethodError.
As it turns out, three of the bundled CPEPs are depended upon by internals so
we cannot move them out of hbase-client/hbase-server. The three modules are
Authentication (used by rpc), MulitRowMutation (used to update meta), and
AccessControl. I worked on refactoring these three CPEPs that internals depend
on, redoing as base functionality so they are no longer CPEPs. Would mean that
changing ACLs would require you to use an hbase-2.0.0 client. Multi was easy
enough to do as an added internal PB Interface addition. Auth seems fine since
it is not actually a CPEP though cast as one. TO FIX. AC is an issue since its
PBs are referenced internally by Query and Mutation...TODO.
I was able to build the shaded hbase-protocol and then have hbase-client and
hbase-server depend on it. Tests would run if I added an explicit dependency
upon hbase-protocol using the ‘system’ label with a path to the built
hbase-protocol module jar (so we picked up shaded artifacts at test time since
they are only available in the hbase-protocol jar, post-package). TODO: Make
this better.
356 <artifactId>hbase-protocol</artifactId>
357 <scope>system</scope>
358
<systemPath>/Users/stack/.m2/repository/org/apache/hbase/hbase-protocol/2.0.0-SNAPSHOT/hbase-protocol-2.0.0-SNAPSHOT.jar</systemPath>
359 </dependency>
Hbase-endpoints picks up a lot of tests moved here from the hbase-server.
Problem is that when the minihbase cluster starts up, it goes to register
supported Interfaces and picks up the local ClientProtos rather than the shaded
ones from hbase-protocol jar referenced against using ‘system’.
420 Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService.newReflectiveBlockingService(Lorg/apache/hadoop/hbase/protobuf/generated/ClientProtos$ClientService$BlockingInterface;)Lorg/
apache/hadoop/hbase/shaded/com/google/protobuf/BlockingService;
421 at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getServices(RSRpcServices.java:1277)
422 at
org.apache.hadoop.hbase.master.MasterRpcServices.getServices(MasterRpcServices.java:214)
423 at
org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:1065)
424 at
org.apache.hadoop.hbase.master.MasterRpcServices.<init>(MasterRpcServices.java:132)
425 at
org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:567)
426 at
org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:557)
427 at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:408)
428 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
429 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
430 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
431 at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
432 at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
433 ... 37 more
TODO: Tests of old school endpoints to ensure they still work.
PROBLEM: IDE Need to remove hbase-protocol and then add dependency on built
hbase-protocol.
PROBLEM: @Ignore @Test // FIX THIS? BROKE AS PART OF SHADING PROJECT!!
public void testCoDelScheduling() throws Exception {
PROBLEM:
TestDefaultCompactSelection.testCompactionRatio:94->TestCompactionPolicy.compactEquals:185->TestCompactionPolicy.compactEquals:204
expected:<[[50, 25, 12, 12]]> but was:<[[]]>
@Ignore @Test // TODO: FIXEEEE
public void testCompactionRatio() throws IOException {
> Figure how to deal with endpoints and shaded pb
> -----------------------------------------------
>
> Key: HBASE-16264
> URL: https://issues.apache.org/jira/browse/HBASE-16264
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, Protobufs
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch,
> HBASE-16264.master.002.patch, HBASE-16264.master.003.patch,
> HBASE-16264.master.004.patch, HBASE-16264.master.005.patch,
> HBASE-16264.master.006.patch, HBASE-16264.master.007.patch
>
>
> Come up w/ a migration plan for coprocessor endpoints when our pb is shaded.
> Would be sweet if could make it so all just worked. At worst, come up w/ a
> prescription for how to migrate existing CPs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
