subject:"\[jira\] \[Commented\] \(HBASE\-15638\) Shade protobuf"

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-12 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570791#comment-15570791
 ] 

Hudson commented on HBASE-15638:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1776 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1776/])
HBASE-16808 Included the generated, shaded java files from HBASE-15638 (stack: 
rev c6289f197685012eeb63b880f151499a47485e2d)
* (edit) hbase-assembly/src/main/assembly/src.xml


> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-05 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547963#comment-15547963
 ] 

Anoop Sam John commented on HBASE-15638:


HBASE-16769

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-05 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547841#comment-15547841
 ] 

stack commented on HBASE-15638:
---

Open a blocker against 2.0 [~anoop.hbase]? Then it'll get attention?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-05 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547800#comment-15547800
 ] 

Anoop Sam John commented on HBASE-15638:


Yes.  For Admin public APIs what we did is we deprecated old PB object methods 
APIs in branch-1 and removed in trunk.  Ya the deprecation happened in the mid 
of 1.x line only.  The argument to relax the compatibility guarantee was that 
Admin is marked evolving.
@InterfaceAudience.Public
@InterfaceStability.Evolving
public interface Admin extends Abortable, Closeable 

Am not sure for CP how we can do now.. Same way of deprecation model as done 
for Admin or deprecate in 2.0 and remove in 3.0?

Ping [~enis]

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-05 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547779#comment-15547779
 ] 

stack commented on HBASE-15638:
---

But we need to do master cp still?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-04 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547772#comment-15547772
 ] 

ramkrishna.s.vasudevan commented on HBASE-15638:


[~saint@gmail.com]
Able to fix the IDE also after this checkin as per your doc. Great. 
REg CP compatability - yes we can change it to use POJO. But when we did the 
Client side APIs it was suggested to add @deprecated tag for the branch-1 APIs 
and that is how it was handled.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-04 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546084#comment-15546084
 ] 

Anoop Sam John commented on HBASE-15638:


I mean the Backward compatibility break as we change the method signature. 
Actually we should change these hooks so as to pass our POJOs.  This way of fix 
was done for public client APIs in Admin etc. But CP hooks missed.  So we 
deprecate the current and add new ones with our own POJOs?  Or is it ok to 
break? how abt CP hook BC guarantee with major version?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-04 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545593#comment-15545593
 ] 

stack commented on HBASE-15638:
---

Just MasterObserver has this issue? If just MasterObserver, I could move this 
back to non-shaded and do a deprecation cycle.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-04 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545589#comment-15545589
 ] 

stack commented on HBASE-15638:
---

How are they broken? [~anoop.hbase]

You mean, this kind of change in MasterObserver where we go from non-shaded to 
shaded types?

{code}
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
@@ -40,8 +40,8 @@ import org.apache.hadoop.hbase.client.MasterSwitchType;
 import org.apache.hadoop.hbase.master.RegionPlan;
 import org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv;
 import org.apache.hadoop.hbase.procedure2.ProcedureExecutor;
-import 
org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.SnapshotDescription;
-import org.apache.hadoop.hbase.protobuf.generated.QuotaProtos.Quotas;
+import 
org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.SnapshotDescription;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Quotas;
{code}



> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-04 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544862#comment-15544862
 ] 

Anoop Sam John commented on HBASE-15638:


We can fix raising a new jira. I can do that based on the decision here.
See also HBASE-15607
Ping [~ram_krish]

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-04 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544854#comment-15544854
 ] 

Anoop Sam John commented on HBASE-15638:


[~saint@gmail.com]
Some of the CP method signatures are changed now after this commit. Is it ok to 
break the CP hooks for 2.0?   We are leaking some of the PB objects (Our proto 
generated) to CPs.  That was not corrected. So this jira changed it to be from 
shaded path.  Any way it is a break. So if break is possible we better correct 
it passing our own pojos rather than PB's
Methods are
{code}
MasterObserver
preListSnapshot
postListSnapshot
preSnapshot
postSnapshot
preCloneSnapshot
postCloneSnapshot
preRestoreSnapshot
postRestoreSnapshot
preDeleteSnapshot
postDeleteSnapshot

preSetUserQuota
postSetUserQuota
preSetUserQuota
postSetUserQuota
preSetUserQuota
postSetUserQuota
preSetTableQuota
postSetTableQuota
preSetNamespaceQuota
postSetNamespaceQuota

RegionServerObserver
preReplicateLogEntries
postReplicateLogEntries
{code}

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-04 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544765#comment-15544765
 ] 

Hudson commented on HBASE-15638:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1723 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1723/])
HBASE-15638 Shade protobuf Which includes (stack: rev 
95c1dc93fb6780f66fd8ebb57914a050b75b9b11)
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/RPCProtos.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/FSProtos.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* (delete) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpointNullResponse.java
* (delete) hbase-protocol/src/main/protobuf/ColumnAggregationProtocol.proto
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SnapshotSentinel.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedScannerCallable.java
* (delete) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetryingCallableBase.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteString.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenInitializing.java
* (add) hbase-protocol-shaded/src/main/protobuf/ClusterId.proto
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CloneSnapshotProcedure.java
* (edit) 
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/UnknownFieldSetLite.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* (add) hbase-protocol-shaded/src/main/protobuf/RegionNormalizer.proto
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Region.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/FloatValue.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/NoOpRetryingInterceptorContext.java
* (delete) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java
* (add) hbase-protocol-shaded/src/main/protobuf/LoadBalancer.proto
* (delete) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/GeneratedMessage.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/TestInterfaceAudienceAnnotations.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/FieldMaskProto.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/IntArrayList.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/LazyField.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ClusterIdProtos.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* (add) hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
* (add) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/compiler/PluginProtos.java
* (add)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-10-03 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544418#comment-15544418
 ] 

stack commented on HBASE-15638:
---

The above set of patches passed the jenkins build I'd made for them: 
https://builds.apache.org/job/HBASE-16264/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/31/
 

Let me now undo this jenkins job.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-09-30 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537173#comment-15537173
 ] 

stack commented on HBASE-15638:
---

HBASE-16741 subtask is about enhancing the build step to add shading and 
pulling in of protobuf lib srcs.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-09-30 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535165#comment-15535165
 ] 

stack commented on HBASE-15638:
---

Update:

 * HBASE-16264 subtask is good to go; does shading of protobuf but makes it so 
CPEPs keep working
 * HBASE-16567, upgrade to 3.1 seems to work. Patch depends on HBASE-16264 
being in first.

Currently working on the checking in shaded protobuf and shaded generated 
source. Trying to elaborate on our current generate-protos step so it does 
generation, shading, and copy-local of the shaded protobuf files. Idea is we 
check this all in and thereafter the build will use the generated sources and 
stuff like IDEs will just work. If we need to patch the protobufs, this will be 
done as part of this -Pcompile-protobuf out-of-band step.

Have my little jenkins build and branch going on to help work through this 
stuff:

https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/
https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=log;h=refs/heads/HBASE-16264





> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-09-20 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508044#comment-15508044
 ] 

stack commented on HBASE-15638:
---

Ok. Have a plan and patch for Endpoints in sub-task. Has downside of 
duplicating .protos; once shaded for core and then non-shaded (for CPEPs). 
Other downsides are temporary -- IDE usage is awkward until we go the next step 
-- or minor (relatively); e.g. we do extra copy carrying CPEP request/response 
in and out of the server.

Next steps are:

 * Checkin of shaded protos; need to add build tooling so the generation and 
shading are easy-to-do out-of-bound with the build.
 * Checkin shaded protobuf source, preferably pb3
 * Checkin our small patches to protobuf to support ByteBuffers; this latter is 
probably our Ram and Anoop.

Let me make subtasks.

Should make a branch? The subtask issue for endpoints is 14.5MB in size. 
Keeping it up is painful because it touches loads of the codebase. I should ask 
out on dev list.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-08-22 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431858#comment-15431858
 ] 

stack commented on HBASE-15638:
---

Oh, [~mbertozzi] has been helping me w/ this stuff.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-08-22 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431838#comment-15431838
 ] 

stack commented on HBASE-15638:
---

Let me update the discussion on dev list.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-08-22 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431832#comment-15431832
 ] 

stack commented on HBASE-15638:
---

I now move to the subtask on CPEPs. It will have the work that is attached here 
as well as what it takes to make the CPEPs work. It should be last issue on the 
shading project.

Review of the posted doc appreciated ([~ghelmling]?). In particular, the added 
pain making your IDE work.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-08-22 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431815#comment-15431815
 ] 

stack commented on HBASE-15638:
---

tl;dr Shading is in hbase-protocol only, not global. Makes it so we can refer 
to both shaded and non-shaded protobuf.  Your IDE is BORKED unless you add 
built hbase-protocol*.jar as runtime dependency. We add in a new module for 
Coprocessor Endpoints called hbase-endpoints; CPEPs will continue to work 
(after a bunch of messing to separate CPEPs from their surrounding context).

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398663#comment-15398663
 ] 

stack commented on HBASE-15638:
---

Ok. Putting this aside for the moment. Let me upload latest. Am going to work 
on purge/narrowing our pb references in a sub issue next.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398483#comment-15398483
 ] 

stack commented on HBASE-15638:
---

The TestInterfaceAudienceAnnotations is interesting. It is shaded pb not being 
available at test time when ClassFinder iterates over classes on CLASSPATH 
(ClassFinder goes to the system classloader explicitly). Messing, I've tried a 
few things: 1. changing the hbase-protocol phase that shading runs at but it 
won't let me... it wants to run at package phase (which makes sense when you 
think about it) which is a phase that comes AFTER test phase; 2. moving this 
test out to hbase-it (Sean idea) only now the CLASSPATH has other pollution in 
it and so a bunch of cleanup of annotations to do in tests BUT we really want 
this class as part of our unit test run because it is great at ensuring code 
gets proper annotations on checkin (it'll fail if missing).

So, I have some hackery in the test that rules out classes that user pb. They 
fail to load with NoClassDefFoundException. That should be ok for stuff like 
CPs. Interesting though, we stumble in a good few places which makes me want to 
revisit my pb cleanup project, the one that narrows the places where we make 
explicit reference to pb. In particular, above I worked on changing 
ServiceException to IOEs but shied from committing this afraid I'd end up 
overcatching. I'm going to redo this project in a separate subissue; talking to 
[~mbertozzi], he pointed out that I went down below Connection where 
ServiceException is thrown, then I could safely make the change/cleanup.

This above hackery brought to the fore what are we to do with our Endpoint CP 
Service; i.e. the subissue HBASE-16264. Let me make comments over there.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-28 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397500#comment-15397500
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 73 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
45s {color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 9s 
{color} | {color:green} master passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 38m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 9m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 5m 
35s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped branch modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 26s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 39s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 49s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 41s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 26s 
{color} | {color:green} master passed with JDK v1.7.0_25 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
49s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 19s 
{color} | {color:red} root in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 19s {color} | 
{color:red} root in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 19s {color} 
| {color:red} root in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 45s 
{color} | {color:red} root in the patch failed with JDK v1.7.0_25. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 45s {color} | 
{color:red} root in the patch failed with JDK v1.7.0_25. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 26m 36s 
{color} | {color:red} hbase-common-jdk1.7.0_25 with JDK v1.7.0_25 generated 1 
new + 25 unchanged - 1 fixed = 26 total (was 26) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 45s {color} 
| {color:red} root in the patch failed with JDK v1.7.0_25. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 37m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 8m 
59s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
45m 40s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397038#comment-15397038
 ] 

stack commented on HBASE-15638:
---

Few of the test failures are interesting class loading problems  Fixed all 
of the javadoc pb references (i think). All the findbugs issues should be done 
too. Try jenkins.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-27 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396937#comment-15396937
 ] 

stack commented on HBASE-15638:
---

Thank you [~ram_krish] for the review.

bq. The change from (InvalidProtocolBufferException e) to IOException is to 
avoid the shading of the 'Exception' classes in protobuf?

No. It is so there are fewer classes with direct pb references. I looked at 
each InvalidProtocolBufferException usage and it wraps a parse of a pb.  
InvalidProtocolBufferException is an IOE. Seemed ok making this change to me.

bq. Some changes in RSGroupManager - was intended to be part of this JIRA where 
the keySet is now changed to EntrySet. 

Yeah. I fixed extant findbugs complaints (these were perf findbugs complaints). 
Trying to get a green run.

Thanks for the review boss.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-27 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396908#comment-15396908
 ] 

ramkrishna.s.vasudevan commented on HBASE-15638:


Went thro the review board. Last 2 pages or so are with test cases. 
The change from (InvalidProtocolBufferException e) to IOException is to avoid 
the shading of the 'Exception' classes in protobuf?
Yes BulkDeletePRotos is back now though the .proto file was always there. 
Some changes in RSGroupManager - was intended to be part of this JIRA where the 
keySet is now changed to EntrySet. 
Rest looks good to me. Thanks Stack.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-26 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394727#comment-15394727
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 73 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 43s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 14s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m 
54s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 20s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 49s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 20s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 0s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 26s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 17s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 17s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 54s 
{color} | {color:red} hbase-common-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 
new + 25 unchanged - 1 fixed = 26 total (was 26) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
16s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
41m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m 
16s {color} | {color:green} the patch passed {color} |
|

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-26 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394466#comment-15394466
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 73 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 59s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 37s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m 
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
49s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 45s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 56s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 7s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 21s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 20s 
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 36s 
{color} | {color:red} hbase-examples in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s {color} 
| {color:red} hbase-examples in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 21s 
{color} | {color:red} hbase-examples in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 20m 33s 
{color} | {color:red} hbase-common-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 
new + 25 unchanged - 1 fixed = 26 total (was 26) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s {color} 
| {color:red} hbase-examples in the patch failed with JDK v1.7.0_80. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 32m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m 
22s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 3s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 53s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 55s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 5s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 15m 16s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red}

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-26 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394133#comment-15394133
 ] 

stack commented on HBASE-15638:
---

Ok. v6 should be better. Includes a commit message describing content too. 
Chasing down all pb references and getting all under shade is a bit of cat 
herding. BulkDelete.proto seems to have gone missing too in our recent history 
but this patch does restore. Lets see how it does.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-26 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393542#comment-15393542
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 72 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 3s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 37s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 38s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 33m 
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
41s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 29s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 7s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 21s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s 
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 26s 
{color} | {color:red} hbase-it in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s 
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 57s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 37s 
{color} | {color:red} hbase-rsgroup in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-it in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s 
{color} | {color:red} hbase-examples in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 57s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s {color} 
| {color:red} hbase-rsgroup in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 40s {color} 
| {color:red} hbase-it in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s {color} 
| {color:red} hbase-examples in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 47s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s 
{color} | {color:red} hbase-rsgroup in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-it in the patch failed with JDK v1.7.0_80. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 24s 
{color} | {color:red} hbase-examples in

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-23 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390894#comment-15390894
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 72 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 51s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 10s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 7s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 25m 
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
28s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 38s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 36s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s 
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 26s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s 
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s 
{color} | {color:red} hbase-it in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 16s 
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 17s 
{color} | {color:red} hbase-rest in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 24s 
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 30s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 22s 
{color} | {color:red} hbase-rsgroup in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 27s 
{color} | {color:red} hbase-it in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 24s 
{color} | {color:red} hbase-examples in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s 
{color} | {color:red} hbase-rest in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 24s {color} 
| {color:red} hbase-client in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 30s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 22s {color} 
| {color:red} hbase-rsgroup in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 27s {color} 
| {color:red} hbase-it in the patch failed with JDK v1.8.0.

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-23 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390879#comment-15390879
 ] 

stack commented on HBASE-15638:
---

My fault. I had not set 'submit patch'

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-22 Thread Vivek Koppuru (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389898#comment-15389898
 ] 

Vivek Koppuru commented on HBASE-15638:
---

Sorry about the comments, I accidentally put my work into the wrong JIRA.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.003.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-21 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388192#comment-15388192
 ] 

stack commented on HBASE-15638:
---

Jenkins is struggling. Let me redo the last patch now HBASE-16263 has been 
applied. My last 'optimization' where I dumb down the catch block for 
ServiceException was not a good idea on second thoughts... let me fix that.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-21 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387466#comment-15387466
 ] 

ramkrishna.s.vasudevan commented on HBASE-15638:


Thanks [~saint@gmail.com]

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-21 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387464#comment-15387464
 ] 

ramkrishna.s.vasudevan commented on HBASE-15638:


It applies to the new patch also. 

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-21 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387226#comment-15387226
 ] 

stack commented on HBASE-15638:
---

Is the +1 for the included subtask HBASE-16263 too? Does it apply to the new 
patch also?  Thanks [~ramkrishna.s.vasude...@gmail.com]

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-21 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387225#comment-15387225
 ] 

stack commented on HBASE-15638:
---

Yessir [~ram_krish].

I just uploaded a new version of the patch. It purges reference to pb 
altogether in a bunch of places. Should help.

A new pb will be fine. We'll do the work to change the src. Tomorrow I'll work 
on the cp endpoint and ensuring that we don't include a protobuf.jar in our end 
product build.

Thanks.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-20 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387187#comment-15387187
 ] 

ramkrishna.s.vasudevan commented on HBASE-15638:


+1. Thanks Stack. So the only thing pending is the Endpoint break? 
Also once we are moving to say a newer version of protobuf and the proto file 
have to be regenerated still we break src compatibility but wire compatibility 
is fine. 

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-20 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387143#comment-15387143
 ] 

stack commented on HBASE-15638:
---

Uploaded patch that changes all pb references from com.google.protobuf to 
org.apache.hbase.shaded.com.google.protobuf. Includes "HBASE-16263 Move all to 
do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol" 
so we compile still.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-20 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386857#comment-15386857
 ] 

stack commented on HBASE-15638:
---

Thank you [~ghelmling] That helps. I think best thing for me to do is try an 
endpoint and see how it breaks.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-20 Thread Gary Helmling (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386820#comment-15386820
 ] 

Gary Helmling commented on HBASE-15638:
---

[~saint@gmail.com], I'm not sure how to make this work with coprocessor 
endpoints.

For endpoints, the user will have:
# Created a .proto file with the service definition for their endpoint
# Run their own version of protoc to do the code gen for the service
# The generated PB messages for request/response, and, as you note, the 
RpcController and RpcCallback in the service method signatures will be in the 
com.google.protobuf package
# The endpoint jar is loaded in HBase
# The ServerRpcController we pass through for the endpoint invocation will 
extend our shaded RpcController

So the only way I see this working is if we force endpoint implementors to run 
a preprocessor step to shade the endpoint jar prior to step #4.  Maybe that 
would work, but it seems pretty cumbersome.  Or else we somehow avoid shading 
ServerRpcController.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-20 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386794#comment-15386794
 ] 

stack commented on HBASE-15638:
---

Made a subtask out of moving all protos under hbase-protocol.

Chatted w/ [~busbey] on approach. He noted that an outcome is that our build 
should not have a protobuf.jar in it at all.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385343#comment-15385343
 ] 

stack commented on HBASE-15638:
---

Stuff to move out of hbase-server and back into hbase-protocol if we go this 
route:

{code}
hbase-server/src/test/protobuf/ColumnAggregationNullResponseProtocol.proto
hbase-server/src/test/protobuf/ColumnAggregationProtocol.proto
hbase-server/src/test/protobuf/ColumnAggregationWithErrorsProtocol.proto
hbase-server/src/test/protobuf/DummyRegionServerEndpoint.proto
hbase-server/src/test/protobuf/IncrementCounterProcessor.proto
hbase-server/src/test/protobuf/PingProtocol.proto
hbase-server/src/test/protobuf/test.proto
hbase-server/src/test/protobuf/test_rpc_service.proto
hbase-server/src/test/protobuf/TestProcedure.proto
{code}

Need to do something w/ these hbase-examples protos:

{code}
hbase-examples/src/main/protobuf/BulkDelete.proto
hbase-examples/src/main/protobuf/Examples.proto
{code}

And these from hbase-rest:

{code}
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/CellSetMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ColumnSchemaMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/NamespacePropertiesMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/NamespacesMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/ScannerMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/StorageClusterStatusMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableInfoMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableListMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/TableSchemaMessage.proto
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/VersionMessage.proto
{code}



> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385114#comment-15385114
 ] 

stack commented on HBASE-15638:
---

Reassessing your suggestion @busbey:

bq. if all references to our internal use of protobuf are in the hbase-protocol 
module, then we needn't have any references to the relocated packages, because 
we can have the shade plugin take care of rewriting them just in that module 
while including the relocated classes that we use within the jar.

We can't do this. pb exceptions are throughout the codebase (could clean these 
up) as is the ByteString type and ipc is doing direct pb Type slinging. Would 
take a good bit of disruptive work looking at the size of the last patch 
attached.


bq. If we go the route of a module that relocated protobuf, we could make a 
profile in our project / top level pom "uses-relocated-protobuf". For those 
modules that activate it, we can then use the shade plugin to rewrite the 
references from the original protobuf to the shaded one. 

We could go this route but I think the profile on certain modules would be 
exotic and then we have to move out the hdfs pb references anyways.

Would you have a few minutes to chat on the above just so I'm sure I am clear 
on what you were suggesting? Thanks [~busbey]


> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-07-09 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369213#comment-15369213
 ] 

Andrew Purtell commented on HBASE-15638:


We should look at this again in light of HADOOP-13363

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242485#comment-15242485
 ] 

stack commented on HBASE-15638:
---

Grand

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242463#comment-15242463
 ] 

Duo Zhang commented on HBASE-15638:
---

OK, I got your point. There would be some maven tricks. We need to pull in 
pb3.0 in hbase-protocol and do not propagate it to other sub-modules since we 
have shaded it in the hbase-protocol artifact. I can help testing whether it 
works in eclipse.

Thanks.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242459#comment-15242459
 ] 

stack commented on HBASE-15638:
---

Thanks [~busbey] Let me look at where we are making direct reference to 
com.google.protobuf outside of hbase-protocol (Apart from the protos in rest 
and spark, I think its rpc that is the culprit ). Let me see what I can 
move back. Let me see how far I get.

On the uses-relocated-protobuf suggestion, let me see list of items we'd need 
to disentangle and what would be involved. My first reaction is that  profile 
for some modules might be a little opaque in its workings but let me turn it 
over as I bang my head here.

Thanks.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242452#comment-15242452
 ] 

stack commented on HBASE-15638:
---

shade happens at the package stage... which is before unit test and the 
hbase-server depends on hbase-protocol so, ok?

For asyncwal calling an HDFS method that expects a PB2.5 ByteString, I think we 
are good. We refer to com.google.protobuf.BS explicity and we'll use the 
transitively included BS that was brought in by the hdfs jar (I think this will 
work).

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242447#comment-15242447
 ] 

Duo Zhang commented on HBASE-15638:
---

Yes, let's only reference the classes in com.google.protobuf in hbase-protocol. 
The asyncwal is an exception and the only problem maybe that the unit tests 
will be broken if the surefire task is executed before shade task? And I'm sure 
that in this way you can not execute unit test in the stupid eclipse IDE... So 
this maybe an advantage of change the import directly in source? To be 
compatible with some IDEs...

Thanks.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242448#comment-15242448
 ] 

Sean Busbey commented on HBASE-15638:
-

{quote}
bq. ... then turning that into a shaded module with relocated protobufs in 
place is pretty straightforward and lets the source reference the original 
names.
Pardon me. Isn't that what this patch is doing?
{quote}

both of the patches posted have references to the shaded protobuf classes in 
e.g. hbase-client. That's the same fall out wether we do a dedicated artifact 
for the shaded protobuf version or roll it into hbase-protocol.

if all references to our internal use of protobuf are in the hbase-protocol 
module, then we needn't have any references to the relocated packages, because 
we can have the shade plugin take care of rewriting them just in that module 
while including the relocated classes that we use within the jar.

If we go the route of a module that relocated protobuf, we could make a profile 
in our project / top level pom "uses-relocated-protobuf". For those modules 
that activate it, we can then use the shade plugin to rewrite the references 
from the original protobuf to the shaded one. It would avoid moving classes 
into hbase-protocol and let us reference the original classes in source, but 
we'd need to move anything that has to deal with HDFS' protobuf out of modules 
that activate the profile.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242440#comment-15242440
 ] 

Sean Busbey commented on HBASE-15638:
-

sorry, I wasn't careful with my language in the prior message. it should have 
been "our and only our" protobuf in the hbase-protocol module

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242439#comment-15242439
 ] 

Sean Busbey commented on HBASE-15638:
-

I thought we were stuck with a reference to the HDFS protobuf in hbase-protocol 
still?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242437#comment-15242437
 ] 

stack commented on HBASE-15638:
---

bq. ... then turning that into a shaded module with relocated protobufs in 
place is pretty straightforward and lets the source reference the original 
names.

Pardon me. Isn't that what this patch is doing?

Let me move the rest and spark module protos back into hbase-protocol.


> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242433#comment-15242433
 ] 

Sean Busbey commented on HBASE-15638:
-

it'll be awkward but makes it clear in the source when we mean to refer to 
"our" version of protobuf vs some other project's version. It also keeps us 
from messing with any of our deployment dependnecies.

on the other hand, if all of the references for "our" protobuf are in the 
hbase-protocol module, then turning that into a shaded module with relocated 
protobufs in place is pretty straightforward and lets the source reference the 
original names.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242422#comment-15242422
 ] 

stack commented on HBASE-15638:
---

Thanks [~busbey]

That mean, you think this patche's approach where all modules explicitly 
reference the relocated pb is the way to go?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242420#comment-15242420
 ] 

stack commented on HBASE-15638:
---

I don't think I can do this [~sergey.soldatov] For sure the later PB version of 
BS is different. I think there could be interesting issues passing a later BS 
to HDFS than it expects.

When I do top-level relocation, which jar ends up w/ the relocated pb classes? 
I can try it and find out for myself...

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242278#comment-15242278
 ] 

Duo Zhang commented on HBASE-15638:
---

I mean we can use the shade plugin when building artifacts without changing the 
source code?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241822#comment-15241822
 ] 

Sean Busbey commented on HBASE-15638:
-

we can't rely on shading doing relocation within the hadoop classes with our 
current deployment instructions. we expressly tell downstream folks to replace 
hte hadoop jars we ship with ones from their hadoop installation.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241687#comment-15241687
 ] 

Sergey Soldatov commented on HBASE-15638:
-

The simple way is just to exclude ByteString from the relocations. In other 
hands if the hdfs classes that are using it is in the final artifact, shading 
plugin is supposed to update them as well. 

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241606#comment-15241606
 ] 

stack commented on HBASE-15638:
---

I see. Let me try your suggestion [~sergey.soldatov] How do I deal with the 
asyncwal hbase case if I'm going a global relocation? I'd exclude the 
problematic asyncwal class?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241576#comment-15241576
 ] 

Sergey Soldatov commented on HBASE-15638:
-

Well, for me the approach to change sources to use shaded packages instead of 
making shading on the final artifact creation is questionable. It makes source 
code navigation painful.  And does it works well with IDEs with a simple 
scenario like load sources into IDE and run unit tests there. I have some 
doubts that it would work without additional manipulations like building 
protocol package first and publish it into local repository. 

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241419#comment-15241419
 ] 

stack commented on HBASE-15638:
---

[~Apache9]
I don't follow. I am using the maven shade plugin in 
https://issues.apache.org/jira/secure/attachment/12798549/15638v2.patch#file-91

Any other input on the approach taken here [~sergey.soldatov] or [~busbey]? If 
not, will try and finish up by moving all protos to hbase-protocol rather than 
have entries in spark and rest as we do now.



> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241414#comment-15241414
 ] 

stack commented on HBASE-15638:
---

I think we can work on this in a different issue. asyncwal no longer presents 
an issue for this effort.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240483#comment-15240483
 ] 

Duo Zhang commented on HBASE-15638:
---

And I suppose the patch here is only used for testing? The shading will be done 
with the maven-shade-plugin right?

Thanks.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240437#comment-15240437
 ] 

Duo Zhang commented on HBASE-15638:
---

Oh I mean a sub-module... Just like 'hbase-common', maybe 'hbase-asyncout'?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239820#comment-15239820
 ] 

stack commented on HBASE-15638:
---

The substitutions I'm running:

{code}
  802  find . -name *.java -exec perl -pi -e 's/import 
com.google.protobuf/import org.apache.hbase.shaded.com.google.protobuf/g' {} \;
  803  find . -name *.java -exec perl -pi -e 's/ com.google.protobuf/ 
org.apache.hbase.shaded.com.google.protobuf/g' {} \;
  804  find . -name *.java -exec perl -pi -e 
's/\(com.google.protobuf/(org.apache.hbase.shaded.com.google.protobuf/g' {} \;
{code}

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239664#comment-15239664
 ] 

stack commented on HBASE-15638:
---

bq. Well, that depends on whether HDFS dependency it are included in the final 
artifact or not. If it is, than shaded version will be called. Otherwise you 
may just exclude a single class from the relocation.

It will be included (so we can talk to hdfs). I could do the transitive include 
of 2.5 pb. We'd have pb2.5 and the shaded pb3 on our classpath. That'd be fine 
I think.  Let me try

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239659#comment-15239659
 ] 

stack commented on HBASE-15638:
---

bq. we could make a module that creates a shaded/relocated version of the 
protobuf version we want to use. Then we could use those relocated packages for 
our internal use.

Is that what this patch does [~busbey]? hbase-protocol is the only place we 
have a dependency on protobuf and it does the relocation that downstream 
modules then make use of.

bq. That way, FanOutOneBlockAsyncDFSOutputSaslHelper can use the protobuf 
version from hadoop like normal. 

Hmm... Thanks for this. The 'fix' then would be to use the pb that hdfs has 
pulled in... which would be com.google.protobuf. That'd work.

bq. It's probably still worth putting it in a module, just so the code is more 
isolated (presuming we eventually want it in HDFS).

I could do this. I always thought that we could have a WAL module... but we'd 
be talking of an explicit asyncwal module in this case?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239497#comment-15239497
 ] 

Sergey Soldatov commented on HBASE-15638:
-

{qoute}
So, shade in the top-level pom?
{quote}
Yep.
{quote}
How will it resolve the ByteString issue? The API being called is from HDFS... 
HDFS has not been shaded. It wants com.google.pb.ByteString but we are passing 
it org.apache.hbase.shaded.com.google.pb.ByteString.
{quote}
Ah. I didn't notice that {{setPayload}} method returns ByteString. Well, that  
depends on whether  HDFS dependency it are included in the final artifact or 
not. If it is, than  shaded version will be called. Otherwise you may just 
exclude a single class from the relocation.  

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239441#comment-15239441
 ] 

Sean Busbey commented on HBASE-15638:
-

we could make a module that creates a shaded/relocated version of the protobuf 
version we want to use. Then we could use those relocated packages for our 
internal use.

That way, FanOutOneBlockAsyncDFSOutputSaslHelper can use the protobuf version 
from hadoop like normal. It's probably still worth putting it in a module, just 
so the code is more isolated (presuming we eventually want it in HDFS).

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239421#comment-15239421
 ] 

stack commented on HBASE-15638:
---

Thank you for taking a look [~sergey.soldatov] 

bq. What is the reason to do it in this way?

Totally open to any approach. My thought was to localize all shading and pb 
machinations inside our hbase-protocol module. Encapsulate the mess.

bq. IMHO it's better to apply the shading pattern to all required modules

So, shade in the top-level pom?

How will it resolve the ByteString issue? The API being called is from HDFS... 
HDFS has not been shaded. It wants com.google.pb.ByteString but we are passing 
it org.apache.hbase.shaded.com.google.pb.ByteString.

Thanks for the help here [~sergey.soldatov]

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239404#comment-15239404
 ] 

stack commented on HBASE-15638:
---

When you say 'new project', you mean a new standalone project up on github or 
something that hbase would pull in?

bq. Can this solve your problem? stack.

I was thinking of just pulling in the hdfs proto Will look again soon. Was 
going to take my time returning to this because was hoping others would have 
better ideas than I (e.g. [~busbey] 'maven tricks' or [~sergey.soldatov]'s 
suggestion above).

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-12 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238637#comment-15238637
 ] 

Duo Zhang commented on HBASE-15638:
---

Let's move FanOutOneBlockAsyncDFSOutputSaslHelper and related classes to a new 
project and do not apply the protobuf shade relocation to it?

Can this solve your problem? [~stack].

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-12 Thread Sergey Soldatov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238343#comment-15238343
 ] 

Sergey Soldatov commented on HBASE-15638:
-

[~stack] What is the reason to do it in this way? IMHO it's better to apply the 
shading pattern to all required modules
Let's say we have modules:
external ModuleA which is suppose to be shaded. (Protobuf)
ModuleB depends on ModuleA (protocols)
ModuleC depends on ModuleB and directly uses some classes from ModuleA (Client)

So, we add ModuleA to the direct dependencies of ModuleC (so it will depends on 
both ModuleB and ModuleA and it doesn't matter whether we have jar for ModuleB 
shaded or not)
And when we build ModuleB and ModuleC we apply the same shading pattern 
"ModuleA->shaded.ModuleA".
That will resolve the problem with ByteString and no need for changing imports. 

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

2016-04-12 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238283#comment-15238283
 ] 

stack commented on HBASE-15638:
---

This depends on HBASE-15174

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

77 matches

Mail list logo