[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721734#comment-15721734 ] Hudson commented on HBASE-14882: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2076 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2076/]) HBASE-14882: Provide a Put API that adds the provided family, qualifier, (anoopsamjohn: rev 004f0abb461cc3ef704824568256bab84b1aa8e7) * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/IndividualBytesFieldCell.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * (add) hbase-common/src/test/java/org/apache/hadoop/hbase/TestIndividualBytesFieldCell.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestPut.java > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch, > HBASE-14882.master.005.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721667#comment-15721667 ] Xiang Li commented on HBASE-14882: -- Anoop, many thanks for your guide and patience! Really appreciate it! > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch, > HBASE-14882.master.005.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721480#comment-15721480 ] Anoop Sam John commented on HBASE-14882: bq.When calculating heapOverhead(), I think I can not make it as a whole constant value and make True.. Ya u have fixed part and varying part based on null or not null bq.In write(OutputStream out, boolean withTags), I return getSerializedSize(withTags) directly as the number of bytes written. I saw you calculated len in ValueAndTagRewriteCell' write(), by adding the size together after each write to output stream. This way of calling getSerializedSize is just fine.. Ideally both this method return value and getSerializedSize() return value must match.. So ur way is also very much fine This is great patch.. +1 > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch, > HBASE-14882.master.005.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721156#comment-15721156 ] Hadoop QA commented on HBASE-14882: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 13s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 19s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 29s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 28m 11s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 43m 49s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12841692/HBASE-14882.master.005.patch | | JIRA Issue | HBASE-14882 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2c5e92ce9bae 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / c73180d | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/4786/testReport/ | | modules | C: hbase-common hbase-client U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/4786/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Provide a Put API that a
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721141#comment-15721141 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase] I uploaded the patch 005 for master branch, to address your comments and have some questions 8-) 1. The following changes are made according to your comments 1.1. Update write(OutputStream out, boolean withTags) to avoid local copy on byte[], with reference to ValueAndTagRewriteCell 1.2. Update headOverhead() to (a) Consider array headers in heapOverhead (b) Use TIMESTAMP_TYPE_SIZE as a sum of size of timestamp and type (c) Make FIXED_OVERHEAD as a static final to be calculated when the class is initialized 1.3. Update deepClone() to return a KeyValue object 1.4. Correct the indents and all updated files are checked 1.5. A new JIRA HBASE-17254 is opened to track the possible update for alignment when calculating heapOverhead() 2. Some questions 2.1. When calculating heapOverhead(), I think I can not make it as a whole constant value and make heapOverhead() returns the constant directly. There are 2 parts: the first part is FIXED_OVERHEAD, which could be constant. But the second part, the array headers for all backing byte arrays, I have to calculate them after the instance has been created, because for family, qualifier, value and tags, ClassSize.ARRAY is added if it is not null, while ClassSize.ARRAY is not added if it is null. 2.2. In write(OutputStream out, boolean withTags), I return getSerializedSize(withTags) directly as the number of bytes written. I saw you calculated len in ValueAndTagRewriteCell' write(), by adding the size together after each write to output stream. Your method is the safest way, while it might be more concise if getSerializedSize(withTags) is returned. Do you think it is safe to return getSerializedSize(withTags) directly? Based on my test, it is safe, but I am not sure if there are some conditions I did not cover. Please advice. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch, > HBASE-14882.master.005.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715402#comment-15715402 ] Anoop Sam John commented on HBASE-14882: 1. Pls see ValueAndTagRewriteCell and how this write is handled 2. All these individual component byte[]s are independent byte[]s. And we consider java array object overhead in heapSize(). This is actually overhead only..So we better consider it in overhead. In case of KV#heapOverhead() we don't consider Array size because mostly many KVs share same byte[]. Atleast in server side when we have it. So this was removed to avoid we overestimate heap size and overhead. This was fixed recently. But for this new Cell impl, we must consider the array overheads as part of heapOverhead() 3. Hmm.. align() is ideally required.. Our bad.. missed some places// May be another issue u can raise to fix all possible places? Am sure many a places it might be missing! > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715001#comment-15715001 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase], thanks for the your time and comments! May I ask some questions about your comments? 1. Regarding bq. This extra copy can be avoided easily Sorry that I did not get your idea. Do you mean that there is another function in KeyValueUtil which can help in write() here but does not do the extra copy? Or do you mean to make it as follow(I meant to use the following code, but finally made it to use KeyValueUtil#appendToByteArray() in patch 004) ? {code} public int write(OutputStream out, boolean withTags) throws IOException { // Key length and then value length out.write(Bytes.toBytes(KeyValueUtil.keyLength(this))); out.write(Bytes.toBytes(getValueLength())); // Row length and then row byte array out.write(Bytes.toBytes(getRowLength())); out.write(getRowArray(), getRowOffset(), getRowLength()); // Family length and then family byte array out.write(getFamilyLength()); out.write(getFamilyArray(), getFamilyOffset(), getFamilyLength()); // Qualifier byte array, no qualifier length out.write(getQualifierArray(), getQualifierOffset(), getQualifierLength()); // Timestamp out.write(Bytes.toBytes(getTimestamp())); // Type out.write(getTypeByte()); // Value out.write(getValueArray(), getValueOffset(), getValueLength()); // Tags length and tags byte array if (withTags && getTagsLength() > 0) { // Tags length byte[] bufferForTagsLength = new byte[2]; Bytes.putAsShort(bufferForTagsLength, 0, getTagsLength()); out.write(bufferForTagsLength); // Tags byte array out.write(getTagsArray(), getTagsOffset(), getTagsLength()); } return getSerializedSize(withTags); } {code} 2. Regarding bq. We add size of 5 refs. All are array type. Means we have to include 5 * ClassSize.ARRAY I put 5 * ClassSize.ARRAY when calculating heapSize() (ClassSize.sizeOf() is called), not in heapOverhead(). Do you mean to move the ClassSize.ARRAY into heapSize()? I referred to KeyValue, in which, ClassSize.ARRAY of bytes is included into heapSize(). 3. Regarding heapOverhead() and heapSize() in KeyValue {code} public long heapSize() { long sum = FIXED_OVERHEAD; /* * Deep object overhead for this KV consists of two parts. The first part is the KV object * itself, while the second part is the backing byte[]. We will only count the array overhead * from the byte[] only if this is the first KV in there. */ return ClassSize.align(sum) + (offset == 0 ? ClassSize.sizeOf(bytes, length) // count both length and object overhead : length);// only count the number of bytes } {code} heapOverhead() does not do the alignment(padding), while alignment of overhead is performed in heapSize(). I might have a different idea: heapOverhead should do alignment before it's return, because the space used in alignment can not be used by others. Do you think so? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712007#comment-15712007 ] Anoop Sam John commented on HBASE-14882: Thanks for the continuous effort here. This took much more effort that we initially thought :-) Some comments {code} byte[] buf = new byte[size]; 84 KeyValueUtil.appendToByteArray(this, buf, 0, withTags); 85 86 out.write(buf); {code} This extra copy can be avoided easily. heapOverhead We add size of 5 refs. All are array type. Means we have to include 5 * ClassSize.ARRAY There is KeyValue.TIMESTAMP_TYPE_SIZE just use that and make 2 additions into 1.. And this entire overhead part is going to be constant. Better we can keep in a static final long and just return that. Avoid calc every time deepClone Suggest create a KV format and return it as a KV object this time. While adding to memstore, this is been called. KV is having much lesser heap overhead Pls do proper formatting of code.. I can see getFamilyOffset is not as per the HBAse guidelines > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711216#comment-15711216 ] Hadoop QA commented on HBASE-14882: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 29s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 22s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 43s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 31m 38s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 49m 44s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12841237/HBASE-14882.master.004.patch | | JIRA Issue | HBASE-14882 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6c76de6c1271 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / fb789b3 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/4741/testReport/ | | modules | C: hbase-common hbase-client U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/4741/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Provide a Put API that add
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711187#comment-15711187 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase] I uploaded patch 004 for master and triggered HADOOP-QA. Patch 004 mainly addresses: 1. Implement ExtendedCell (to override 2 write() and getSerializedSize()) 2. Enhance UT. Previously, the expected value of assert was hard-coded. I changed them to verify if IndividualBytesFieldCell has the same behaviors/outputs as KeyValue when given the same inputs. For the implementation of 2 write() and getSerializedSize(), I used the functions in KeyValueUtil as much as I can. I also implemented the logic myself according to KeyValue format, which I think might be a little more efficient, at least no copy is made, and write into byte buffer or output stream directly. The reason why I chose to use the functions in KeyValueUtil is that it saves a lot of efforts of the maintenance in the future, when somebody would like to modify the key value format. He does not need to care about the implementation in IndividualBytesFieldCell, as long as the functions in KeyValueUtil are modified. The drawback is that it might be less efficient than that implemented directly, when calling write(). What is your opinion? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch, HBASE-14882.master.004.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695461#comment-15695461 ] Anoop Sam John commented on HBASE-14882: #1 - Oh ya the state var name might have changed over the time.. Ya it is the heap requirement for having a long state in KV. ie. seqId.. We called it memstoreTS before. #2 - the former write API is used while copying the cell data over the MSLAB. There is no Q we have to avoid writing tags. we must consider tags always. The latter write API used while writing to RPC/HFile. You know, we have Codecs to include tags or not.. So this boolean.. Any way see a TODO there.. We have to even avoid this boolean also.. I may start working on that soon. That is a change needed in how we deal with tags serializing over wire. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695409#comment-15695409 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase], thanks for the explanation! 2 more questions... (1) In KeyValue, you define FIXED_OVERHEAD, as (from line 88) {code} public static final long FIXED_OVERHEAD = ClassSize.OBJECT + // the KeyValue object itself ClassSize.REFERENCE + // pointer to "bytes" 2 * Bytes.SIZEOF_INT + // offset, length Bytes.SIZEOF_LONG;// memstoreTS <-- here {code} For the last line, "Bytes.SIZEOF_LONG;// memstoreTS", does it count the size for seqId ? memstoreTS = seqId = MVCC version? (2) In ExtendedCell, why does the following write() not have a input as withTags? {code} void write(ByteBuffer buf, int offset); {code} While {code} int write(OutputStream out, boolean withTags) {code} and {code} int getSerializedSize(boolean withTags) {code} both allow user to specify whether he/she would like to serialize tags. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692623#comment-15692623 ] Anoop Sam John commented on HBASE-14882: U can use KeyValueUtil#appendToByteArray > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686240#comment-15686240 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase] When implementing ExtendedCell interface for IndividualBytesFieldCell, to override the following function {code} /** * Write the given Cell into the given buf's offset. * @param buf The buffer where to write the Cell. * @param offset The offset within buffer, to write the Cell. */ void write(byte[] buf, int offset); {code} I found KeyValue#createByteArray() (as follow) can be re-used {code} static byte [] createByteArray(final byte [] row, final int roffset, final int rlength, final byte [] family, final int foffset, int flength, final byte [] qualifier, final int qoffset, int qlength, final long timestamp, final Type type, final byte [] value, final int voffset, int vlength, byte[] tags, int tagsOffset, int tagsLength) { {code} So I need to check with you 1. If ExtendedCell#write() (listed above) needs to write the Cell in KeyValue format, 2. If the answer for 1 is yes, then may I propose to make KeyValue#createByteArray() to package-private so that its logic can be used in other ExtendedCell implementations? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15666234#comment-15666234 ] Anoop Sam John commented on HBASE-14882: checkParameters() - Ya u can make it non private. May be package private is enough? checkForTagsLength() - Suggest move this and MAX_TAGS_LENGTH to TagUtil.. That is a better place IMO. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665951#comment-15665951 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase] In keyValue, may I propose to set the following 2 functions (to check parameters) to be public? - checkParameters() - checkForTagsLength() I think those 2 functions can be re-used by IndividualCellBytesFieldCell's constructor to check the parameters, as well as other Cell or ExtendedCell implementations. Does it make sense to you? if so, may I open a new JIRA? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658804#comment-15658804 ] Xiang Li commented on HBASE-14882: -- Hi Anoop, I see, got your idea. Thanks! I will update the patch to (1) Implement ExtendedCell (2) Add some comments to explain why IndividualBytesFieldCell at client end, has ExtendedCell implemented Please correct me if I did not get your idea. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656453#comment-15656453 ] Anoop Sam John commented on HBASE-14882: So now we clearly know, why this API is not client end alone. The code paths within server also uses this API. eg: When adding to meta table, we make a put and add cells using this API. This happens within server end and the cell as such reaches region. We have assumptions abt the cell there. That it is having SettableSeqId implemented etc. So we can not just avoid that. Normal APIs in Put u can see create KeyValue and there u can see impl ExtendedCell instead of Cell. Ya this is because KV is used server end. We need cells flowing in server to be of this new type. As this new Cell impl also used in server side also, we can not avoid that also impl ExtendedCell !! May be need to add fat comment lines in the new class like above that why at client end, we have a Cell with ExtendedCell being used. Any way we have it in hbase-common. That is good. w dyt? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656231#comment-15656231 ] Anoop Sam John commented on HBASE-14882: Sorry missed this some how. Will look at the latest patch today. Thanks. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655802#comment-15655802 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase] Would you please help to review patch 003 when you have time ^_^ > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650466#comment-15650466 ] Hadoop QA commented on HBASE-14882: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 17s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 17s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 25s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 41m 56s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12837942/HBASE-14882.master.003.patch | | JIRA Issue | HBASE-14882 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5225a55e812c 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 03bc884 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/4397/testReport/ | | modules | C: hbase-common hbase-client U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/4397/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Provide a Put API that adds
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15647093#comment-15647093 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase] After testing almost all code paths (sorry it took a long time) and verifying by UT, patch 003 is uploaded for master branch. Please review and comment at your most convenience. Thanks! The following changes are included: 1. In the constructor of IndividualBytesFieldCell, remove the check on qualifier and value. Do not throw IllegalArgumentException if qualifier/value is null. 2. Handle some getters when qualifier/value/tags is null: (1) Make getXXXArray() for qualifier, value and tags return HConstansts.EMPTY_BYTE_ARRAY if qualifier/value/tags is null, and make getXXXLength() for qualifier, value and tags return 0 if qualifier/value/tags is null. Those have the same behaviors as KeyValue. By the way, input row is checked by Mutation#checkRow() in the constructor of Put, and family is checked in the constructor of IndividualBytesFieldCell, so in getXXXArray() and getXXXLength() for row and family, there is no check whether it is null or not. (2) In heapSize(), call getQualifierLength(), getValueLength() and getTagsLength() instead of qualifier/value/tags.length, in case that qualifier/value/tags is null. (3) Add a new @Test called testNullQualifierValueTags to check the output of getXXXArray(), getXXXLength() and getXXXOffset() when qualifier/value/tags is null. 3. Add annotation to indicate the test category which TestIndividualBytesFieldCell belongs to, according to HBASE-11911. *One question:*, HBASE-16134 makes SettableSequenceId and SettableTimestamp deprecated in 2.0 and to be removed in 3.0. Could I still use them in this JIRA if I would like this JIRA get into 2.0? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, > HBASE-14882.master.003.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516752#comment-15516752 ] Xiang Li commented on HBASE-14882: -- The following 2 Put#addImmutable() are updated by the patch {code} public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) {code} and {code} public Put addImmutable(byte [] family, byte [] qualifier, long ts, byte [] value) {code} In master branch, the first one is used: - (master) In MeteTableAccessor, when writing region assignment info into "hbase:meta" table - (master) In RegionStateStore, when updating region state in "hbase:meta" table - (master) In TableNamespaceManager, when creating namespace (insert into "hbase:namespace" table) - (server) In AccessControlLists, when updating "hbase:acl" table - (server) In DefaultVisibilityLabelServiceImpl, when adding system visibility labels - (thrift) In ThriftServerRunner.HBaseHandler and ThriftUtilities - Some UT the second one is used: - (master) In MeteTableAccessor, when writing "hbase:meta" table - (master) In FavoredNodeAssignmentHelper when writing "hbase:meta" table - (rest) In RowResource - (thrift) In ThriftUtilities I am testing those code paths to verify if the two new Put#addImmutable() work and also checking if the inputs to Put#addImmutable() (family, qualifier and value) in those usages are immutable > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509533#comment-15509533 ] Anoop Sam John commented on HBASE-14882: I mean within HBase code base, where all put#addImmutable is being used? Some where in the WAL read code flow it is getting used for sure. That is what the failure u got when u have not made the new Cell to extend SettableSeqId > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509299#comment-15509299 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase], I uploaded patch 002 for master. Please review. 1. There is a warning reported by findbugs, but it has nothing to do with the patch 8-) 2. Regarding your comment bq. Can u see all usage of these affected Put APIs.. (addImmutable) Sorry that I did not get your question. Could you please elaborate more? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503978#comment-15503978 ] Hadoop QA commented on HBASE-14882: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 27s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 19s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 36s {color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 26m 38s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 42m 29s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12829228/HBASE-14882.master.002.patch | | JIRA Issue | HBASE-14882 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4ac2471e3214 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / c5b8aab | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/3604/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/3604/testReport/ | | modules | C: hbase-common hbase-client U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/36
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503868#comment-15503868 ] Xiang Li commented on HBASE-14882: -- Hi Anoop, I re-uploaded patch 002. It failed when I tried to uploaded it just now. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch, HBASE-14882.master.002.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503855#comment-15503855 ] Anoop Sam John commented on HBASE-14882: bq.I uploaded patch version 002 for master, You forgot to attach this version? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503808#comment-15503808 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase], thanks very much for the explanations! I uploaded patch version 002 for master, to address the following changes/comments: 1. The new implementation is extracted out of Put and named as "IndividualBytesFieldCell" and put in the same package as KeyValue. "Put" type is not hard-coded in IndividualBytesFieldCell, but it is inputable for the constructor of IndividualBytesFieldCell. While in Put#addImmutable(), it is set as "Put" type. 2. More interfaces are implemented: HeapSize, Cloneable, SettableSequenceId, SettableTimestamp. (1) For Cloneable, only a shadow copy is made for each backing byte arrays. Does it make sense to you? (2) Streamable is not implemented 3. Add UT for both IndividualBytesFieldCell and Put#addImmutable() Would you please review it, thanks! > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483085#comment-15483085 ] Anoop Sam John commented on HBASE-14882: Thanks for the tests.. 1. Ya ideally those interfaces are not required to be used at client end. Here it seems like Put#addImmutable() API is being used by some WAL replay path. On those cells coming in for replay, we tend to set the seqId. Ya this is fully server end code path only. The client API is used by server also.. So we need to be careful here.. Can u see all usage of these affected Put APIs.. (addImmutable) 2. Streamable is not a must any way. If this is implemented, then the Codec can do an optimized way of serializing this Cell on to wire IN KV FORMAT. So the serialization to wire is always KV format way.. Because of that, the deserialization by codec is simple. It just deserialize into KV. So there are no confusion as u mentioned above > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15476403#comment-15476403 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase], I have some questions, could you please take a look at them when you have time? 1. KeyValue implements 5 more interfaces beside Cell, they are HeapSize, Cloneable, SettableSequenceId, SettableTimestamp, Streamable(Streamable is only in master branch for 2.0.0). Do I need to implement them for IndividualBytesFieldCell? I think SettableSequenceId is a must, because I found an error when testing my patch without SettableSequenceId implemented. When HBase is re-started, the following exception can be found in RS log when master asks the RS to open a region {code} java.io.IOException: java.lang.UnsupportedOperationException: Cell is not of type org.apache.hadoop.hbase.SettableSequenceId at org.apache.hadoop.hbase.CellUtil.setSequenceId(CellUtil.java:676) at org.apache.hadoop.hbase.regionserver.wal.FSWALEntry.stampRegionSequenceId(FSWALEntry.java:124) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1853) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.UnsupportedOperationException: Cell is not of type org.apache.hadoop.hbase.SettableSequenceId ... 9 more {code} SettableSequenceId and SettableTimestamp are server-side code to me, but Put is client-side code. I am just wondering why the Cell implementation used in Put(in client side) are expected to implement some server-side code. 2. About Streamable interface which is added by HBASE-13754. (1) Do I need to implement Streamable as well? Regarding the description of HBASE-13754, you mentioned: bq. For other cell implementation, we will call getXXXLength() and getXXXArray() and write each component one after the other It can improve the efficiency if I have a single backing array as a whole and implement Streamable, but it is still ok if it is not implemented. I will just call getXXXArray(), getXXXLength and getXXXOffset() and write each field one by one if I override Streamable#write(). So it is not a must, do I get it right? (2) I read your discussion with Stack but still has a question for not having a method to do de-serialization (that is, read()) to pair with write(). Regarding your comment in HBASE-13754 when explaining your idea why there is only write() but no read() {quote} I think that is the correct path especially if we add the method to Cell interface itself. All the Cell impls to have default constructor. The Codec will create a Cell object of its choice and ask the object to make its data from the stream. But with the new interface type I thought to just leave it as we are not sure whether all Cells will impl the new interface {quote} Do you mean that if one needs to do the de-serialization against my new Cell implementation - IndividualBytesFieldCell, he/she needs to cut each field from stream and then call the constructor to build a new Cell or a new IndividualBytesFieldCell? But in this way, he/she needs to read and fully understand what I code in IndividualBytesFieldCell#write(). Does it not quite follow the rule of encapsulation and not exposing the internal implementation? I mean, he/she must know clearly about the internal fields of IndividualBytesFieldCell and how I code write(), in order to implement read() correctly. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualif
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470985#comment-15470985 ] Jerry He commented on HBASE-14882: -- +1 on IndividualBytesFieldCell Yes, Better to change the Type field as an imputable as well given the general name and for general usage. But for the PUT API, this field is still fixed. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470913#comment-15470913 ] Anoop Sam John commented on HBASE-14882: This is not a KeyValue kind of structure.. So no ame like that pls. How abt IndividualBytesFieldCell? - A cell whose fields are individual bytes. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470423#comment-15470423 ] Xiang Li commented on HBASE-14882: -- Hi Jerry, Anoop, what about "SeparateKeyFieldKeyValue" ? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469642#comment-15469642 ] Jerry He commented on HBASE-14882: -- Maybe 'ImmutableKvFieldCell'? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469330#comment-15469330 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase], what about "ImmutableKeyFieldCell" or "CellWithImmutableKeyFields" or "SeparateKeyFieldCell" ? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15463870#comment-15463870 ] Anoop Sam John commented on HBASE-14882: [~tedyu], [~enis] Any suggestion for the new class name? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15463868#comment-15463868 ] Anoop Sam John commented on HBASE-14882: InputsImmutableCell -> I dont think this reflect the actual thing. It is not that this cell is made of Immutable bytes is the key.. Even when we create KV also (passing a byte[]) we assume it is immutable. Just refer that.. But there we assume the entire key, value etc are in single contiguous byte[]. Here each component (RK, CF, value etc) of the Cell is separate byte[].. And this class is implementing Cell this way. I believe we need a name in that way. Sorry am not good in naming. :-( Do we need CellUtil API? I guess no. U can keep the new Class (implementing Cell) in same place as of KeyValue. Just create new instance of that within Put#addImmutable. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461258#comment-15461258 ] Xiang Li commented on HBASE-14882: -- [~enis], thanks for the comments! [~anoop.hbase] and [~enis]: The new implementation is extracted out of Put and I put it into CellUtil as a public static inner class of CellUtil, and name it as "InputsImmutableCell". Does it make sense to you? The reason why putting it into CellUtil is that I found there are some Cell implementations inside CellUtil, such as EmptyCell... But it will be placed into a different package (not hbase.client as Put, but hbase) and a different jar (not hbase-client.jar, but hbase-common.jar). Do you think it is reasonable to put it there, or it is better to be placed in hbase.client, to be together with Put? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439189#comment-15439189 ] Hudson commented on HBASE-14882: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1485 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1485/]) HBASE-14882 Provide a Put API that adds the provided family, qualifier, (anoopsamjohn: rev a63ba09c99fdc5f61a251a11987d33672fac1d65) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestPut.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java Revert "HBASE-14882 Provide a Put API that adds the provided family, (anoopsamjohn: rev e0aee7c74f8c75f2b831a682db6e8152e958bfd8) * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestPut.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438876#comment-15438876 ] Anoop Sam John commented on HBASE-14882: Sorry did not see ur comment.. I will revert and reopen it > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438870#comment-15438870 ] Enis Soztutar commented on HBASE-14882: --- Can we please extract the Cell implementation out of the Put, and make it a generic Cell implementation similar to the CellImpl in HBASE-15945. This maybe pretty useful outside of Put on the client side. bq. I was not able to find HBASE-15945 in git log, is it committed?Anyway, I can read the patch file directly _ It is in the C++ branch still, not merged to master yet. The branch is HBASE-14850. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438861#comment-15438861 ] Anoop Sam John commented on HBASE-14882: Thanks for the study and confirm. :-) Ya I was thinking so but wanted to double confirm.. +1 > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438826#comment-15438826 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase] bq. Any implication on class loading of the anonymous inner class? Every time code path reach, need new defn and load? Some updates: 1. (Anonymous) inner class is treated the same as normal class, in terms of class loading and initializing. It will be loaded and initialized when the first time it is used (or pre-load of class is used). So the anonymous inner class will be loaded only once. 2. The instance of inner class has a reference to the instance of outer class. It can bee seen by de-compiling .class file, like: (Put$1 is the anonymous inner class) {code} class Put$1 implements Cell { Put$1(Put this$0, byte[] var2, byte[] var3, long var4, byte[] var6) { ~~ this.this$0 = this$0; this.val$family = var2; this.val$qualifier = var3; this.val$ts = var4; this.val$value = var6; } ... } {code} It will impact GC. The instances of outer class will not be cleaned timely. I see no other impactions so far. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432991#comment-15432991 ] Xiang Li commented on HBASE-14882: -- Patch 001 for master branch is uploaded to address the comment 2 and 5(see my comment above) from Anoop. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch, > HBASE-14882.master.001.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432910#comment-15432910 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase], thanks for the comments! 1. Regarding bq. Why not handling this API ? addImmutable(byte [] family, byte [] qualifier, byte [] value) It internally calls the updated function, as {code} public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) { return addImmutable(family, qualifier, this.ts, value); } {code} So do not need to be updated. 2. Regarding the return value of getTagsArray() Yes, I should return an empty byte array. Updated to use EMPTY_BYTE_ARRAY in HConstants. 3. Regarding bq. All these checks on row, cf copied from KV class? KeyValue has the similar function to check the parameters passed to the constructors. It is called checkParameters(), starting from line 827 of KeyValue, as well as some other places. I add some more check based on it. 4. Regarding bq. Any implication on class loading of the anonymous inner class? Every time code path reach, need new defn and load? I dont think so. Just need to check if I get your idea. Do you worry that it might impact performance to use anonymous inner class here? I see, but I have no idea so far. Will study. 5. Regarding bq. Can have a single if block with || check? Exception can say that the family has to be of length > 0 and < max Thanks for the comments! Addressed. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430104#comment-15430104 ] Anoop Sam John commented on HBASE-14882: Why not handling this API ? addImmutable(byte [] family, byte [] qualifier, byte [] value) {code} public byte[] getTagsArray() { 350 return null; 351 } {code} Better to return empty byte[]. We should be having constants declared for this some where. HConstants? All these checks on row, cf copied from KV class? Any implication on class loading of the anonymous inner class? Every time code path reach, need new defn and load? I dont think so. Just asked for confirm. {code} if (family == null) { 231 throw new IllegalArgumentException("Family is null"); 232 } 233 if (family.length == 0) { // family can not be "" 234 throw new IllegalArgumentException("Family length is 0"); 235 } 236 if (family.length > HConstants.MAX_FAMILY_LENGTH) { 237 throw new IllegalArgumentException("Family length > " + HConstants.MAX_FAMILY_LENGTH); 238 } {code} Can have a single if block with || check? Exception can say that the family has to be of length > 0 and < max > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429564#comment-15429564 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase] Would you please do me a favor to review the patch? Thanks! > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418890#comment-15418890 ] Hadoop QA commented on HBASE-14882: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 23s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 4s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 42s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 10s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 58m 40s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-12 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12823460/HBASE-14882.master.000.patch | | JIRA Issue | HBASE-14882 | | Optional Tests | asflicense javac javad
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418821#comment-15418821 ] Xiang Li commented on HBASE-14882: -- Hi [~anoop.hbase], I uploaded patch 000 for master branch and the same logic has been tested on HBase 1.1.1. The following addImmutable() is modified, with an anonymous Cell used. {code} public Put addImmutable(byte [] family, byte [] qualifier, long ts, byte [] value) {code} Because addImmutable() which is mentioned in the description(as below) internally calls the modified one, so its behavior is changed as well. {code} public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) {code} I did not update addImmutable() which have tag as its input, as they are marked as @InterfaceAudience.Private. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Jerry He >Assignee: Xiang Li > Fix For: 2.0.0, 1.2.2 > > Attachments: HBASE-14882.master.000.patch > > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417487#comment-15417487 ] Xiang Li commented on HBASE-14882: -- Got that, anonymous Cell is a good idea! Thanks stack! > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Xiang Li > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation, the family, qualifier and value are still being > copied locally to create kv. > Hopefully we should provide an API that truly uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408891#comment-15408891 ] Anoop Sam John commented on HBASE-14882: Ya.. we can make the API such that all passed to be immutable so that we can avoid copy. We can refer to value byte[] directly in new Cell imp. Sorry am not good in naming. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Xiang Li > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation the row, family. qualifier and value are still > being copied locally to create kv. > Hopefully we should provide an API that truely uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408861#comment-15408861 ] Xiang Li commented on HBASE-14882: -- [~enis], thanks for the comment! I was not able to find HBASE-15945 in git log, is it committed?Anyway, I can read the patch file directly ^_^ > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Xiang Li > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation the row, family. qualifier and value are still > being copied locally to create kv. > Hopefully we should provide an API that truely uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405566#comment-15405566 ] Xiang Li commented on HBASE-14882: -- [~anoop.hbase], I am drafting the very first patch, something need to discuss with you, as follow: 1. I plan to name the new implementation as "InputsImmutableCellForPut", and put it as an internal protected class within Put class. Sounds good to you? 2. Regarding {code} /** * See {@link #addColumn(byte[], byte[], byte[])}. This version expects * that the underlying arrays won't change. It's intended * for usage internal HBase to and for advanced client applications. */ public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) {code} Do wee need the caller to guarantee value is also immutable as family and qualifier? Can we keep a reference to value in the new implementation directly? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Xiang Li > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation the row, family. qualifier and value are still > being copied locally to create kv. > Hopefully we should provide an API that truely uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403138#comment-15403138 ] Xiang Li commented on HBASE-14882: -- HI Stack, thanks for the comment! Do you mean to use an anonymous internal class which implements Cell and resides in Put? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Xiang Li > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation the row, family. qualifier and value are still > being copied locally to create kv. > Hopefully we should provide an API that truely uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383129#comment-15383129 ] Enis Soztutar commented on HBASE-14882: --- Replicate what we have done for our C++ client in HBASE-15945? Basically a {{CellImpl}} which looks like our PB cell object, individual row, cf, column, value fields. > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Xiang Li > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation the row, family. qualifier and value are still > being copied locally to create kv. > Hopefully we should provide an API that truely uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[ https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383095#comment-15383095 ] stack commented on HBASE-14882: --- Does it need to have a name? Can we not just make an anonymous Cell? > Provide a Put API that adds the provided family, qualifier, value without > copying > - > > Key: HBASE-14882 > URL: https://issues.apache.org/jira/browse/HBASE-14882 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Xiang Li > > In the Put API, we have addImmutable() > {code} > /** >* See {@link #addColumn(byte[], byte[], byte[])}. This version expects >* that the underlying arrays won't change. It's intended >* for usage internal HBase to and for advanced client applications. >*/ > public Put addImmutable(byte [] family, byte [] qualifier, byte [] value) > {code} > But in the implementation the row, family. qualifier and value are still > being copied locally to create kv. > Hopefully we should provide an API that truely uses immutable family, > qualifier and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)