subject:"\[jira\] \[Commented\] \(HBASE\-6200\) KeyComparator.compareWithoutRow can be wrong when families have the same prefix"

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401195#comment-13401195
]

Jieshan Bean commented on HBASE-6200:
-

Ya...You reminded me. I have one idea to optimize this:
1. if (left family length != right family length). Only compare column family
is enough.
2. if (left family length == right family length). we can put them together to
compare.
So no mattter which case, only one comparison will happen.
I will test it right now.

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Key: HBASE-6200
URL: https://issues.apache.org/jira/browse/HBASE-6200
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch,
HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch,
HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch,
HBASE-6200-trunk.patch, PerformanceTestCase-6200-94.patch

As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird
behavior when some families share the same prefix. He posted a link to his
code to show how it fails, http://pastebin.com/7TBA1XGh
Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families
and qualifiers so f:a is said to be bigger than f1:, which is false. Then
what happens is that the KVs are returned in the right order from the RS but
then doing {{Result.binarySearch}} it uses
{{KeyComparator.compareWithoutRow}} which has a different sorting so the end
result is undetermined.
I added some debug and I can see that the data is returned in the right order
but {{Arrays.binarySearch}} returned the wrong KV, which is then verified
agains the passed family and qualifier which fails so null is returned.
I don't know how frequent it is for users to have families with the same
prefix, but those that do have that and that use those families at the same
time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401205#comment-13401205
 ] 

Zhihong Ted Yu commented on HBASE-6200:
---

The above approach should work.

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch, PerformanceTestCase-6200-94.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-26 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401454#comment-13401454
 ] 

stack commented on HBASE-6200:
--

Thanks for taking time on perf Jieshan.  +1 on commit (It looks like you have 
enough tests).

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, 6200-trunk-v3.patch, 
 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-26 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401483#comment-13401483
]

Hadoop QA commented on HBASE-6200:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533487/6200-trunk-v4.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.security.access.TestAccessController

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2260//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2260//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2260//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2260//console

This message is automatically generated.

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Attachments: 6200-trunk-v2.patch, 6200-trunk-v3.patch,
6200-trunk-v4.txt

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401508#comment-13401508
 ] 

Zhihong Ted Yu commented on HBASE-6200:
---

I ran the two tests listed above and they passed.

@Jieshan:
Please prepare patches for 0.94, 0.92 and 0.90 branches based on patch v4.

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, 6200-trunk-v3.patch, 
 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401807#comment-13401807
]

Zhihong Ted Yu commented on HBASE-6200:
---

Patch for 0.94 is same as the patch for trunk.

I ran tests for 0.92 and they passed.

Integrated to trunk, 0.94 and 0.92
Will integrate to 0.90 once the patch is ready.

Thanks for the patch, Jieshan.

Thanks for the review, Stack and Lars.

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-trunk-v2.patch,
6200-trunk-v3.patch, 6200-trunk-v4.txt

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401829#comment-13401829
 ] 

Hudson commented on HBASE-6200:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #70 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/70/])
HBASE-6200 KeyComparator.compareWithoutRow can be wrong when families have 
the same prefix (Jieshan) (Revision 1354290)

 Result = FAILURE
tedyu : 
Files : 
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java


 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-trunk-v2.patch, 
 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401841#comment-13401841
 ] 

Jieshan Bean commented on HBASE-6200:
-

Thank you Ted. I will upload the patch for 90 immediately(I planed to make all 
the patches after review:))..

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-trunk-v2.patch, 
 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401851#comment-13401851
 ] 

Hudson commented on HBASE-6200:
---

Integrated in HBase-0.94 #284 (See 
[https://builds.apache.org/job/HBase-0.94/284/])
HBASE-6200 KeyComparator.compareWithoutRow can be wrong when families have 
the same prefix (Jieshan) (Revision 1354293)

 Result = SUCCESS
tedyu : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java


 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-trunk-v2.patch, 
 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401861#comment-13401861
 ] 

Jieshan Bean commented on HBASE-6200:
-

Sorry.. It was made based on V4 not V2..

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-90.patch, 
 6200-trunk-v2.patch, 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401868#comment-13401868
 ] 

Hudson commented on HBASE-6200:
---

Integrated in HBase-TRUNK #3077 (See 
[https://builds.apache.org/job/HBase-TRUNK/3077/])
HBASE-6200 KeyComparator.compareWithoutRow can be wrong when families have 
the same prefix (Jieshan) (Revision 1354290)

 Result = FAILURE
tedyu : 
Files : 
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java


 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-90.patch, 
 6200-trunk-v2.patch, 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401927#comment-13401927
 ] 

Hudson commented on HBASE-6200:
---

Integrated in HBase-0.92 #459 (See 
[https://builds.apache.org/job/HBase-0.92/459/])
HBASE-6200 KeyComparator.compareWithoutRow can be wrong when families have 
the same prefix (Jieshan) (Revision 1354294)

 Result = FAILURE
tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java


 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-90.patch, 
 6200-trunk-v2.patch, 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401943#comment-13401943
 ] 

Jieshan Bean commented on HBASE-6200:
-

All tests passed for 90.

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-90.patch, 
 6200-trunk-v2.patch, 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401951#comment-13401951
 ] 

Zhihong Ted Yu commented on HBASE-6200:
---

Integrated to 0.90 branch as well.

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-90.patch, 
 6200-trunk-v2.patch, 6200-trunk-v3.patch, 6200-trunk-v4.txt


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400303#comment-13400303
 ] 

Anoop Sam John commented on HBASE-6200:
---

{code}
-  if (lcolumnlength == 0  ltype == Type.Minimum.getCode()) {
+  if (lfamilylength == 0  lqualifierlength == 0
+   ltype == Type.Minimum.getCode()) {
{code}
May be only lfamilylength ==0 check is enough. When the family is missing from 
KV there can not be a qualifier.
Same with
{code}
-  if (rcolumnlength == 0  rtype == Type.Minimum.getCode()) {
+  if (rfamilylength == 0  rqualifierlength == 0
+   rtype == Type.Minimum.getCode()) {
{code}



 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Jieshan Bean (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400323#comment-13400323
 ] 

Jieshan Bean commented on HBASE-6200:
-

@Anoop:
Thanks for your carefully review. 
Only the createLastOnRow methods using Type.Minimum. Likes below:
{noformat}
 public static KeyValue createLastOnRow(final byte[] row) {
return new KeyValue(row, null, null, HConstants.LATEST_TIMESTAMP, 
Type.Minimum);
 }
{noformat}
At this time, the length of CF (or Qualifier) should be 0. 
Actually, we can create one KeyValue with CF as empty but Qualifier is not 
empty. And if we call KeyValue#getQualifier, we can get the correct value. So I 
think this check should be strict with both CF == 0 and Qualifier == 0.


 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400443#comment-13400443
]

Hadoop QA commented on HBASE-6200:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12533286/PerformanceTestCase-6200-94.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

-1 patch. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2249//console

This message is automatically generated.

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400462#comment-13400462
 ] 

Lars Hofhansl commented on HBASE-6200:
--

So if I read that right 1.000.000 KV compares take 100ms without the patch and 
121ms with the patch. I think that would be within the noise of any real 
operation.


 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch, PerformanceTestCase-6200-94.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Zhihong Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400529#comment-13400529
 ] 

Zhihong Ted Yu commented on HBASE-6200:
---

@Lars:
What other tests do you think should be performed ?

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch, PerformanceTestCase-6200-94.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Zhihong Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400698#comment-13400698
 ] 

Zhihong Ted Yu commented on HBASE-6200:
---

@Stack:
Do you have further comments on the latest patches ?

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch, PerformanceTestCase-6200-94.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Jieshan Bean (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401094#comment-13401094
 ] 

Jieshan Bean commented on HBASE-6200:
-

bq.So if I read that right 1.000.000 KV compares take 100ms without the patch 
and 121ms with the patch. 
1,000,000 times called the below method(Will compare 2 times in each call), to 
be exact:
{noformat}
  private void assertKVLessWithoutRow(KeyValue.KeyComparator c, int common, 
KeyValue less,
  KeyValue greater) {
int cmp = c.compareIgnoringPrefix(common, less.getBuffer(), less.getOffset()
+ KeyValue.ROW_OFFSET, less.getKeyLength(), greater.getBuffer(),
greater.getOffset() + KeyValue.ROW_OFFSET, greater.getKeyLength());
assertTrue(cmp  0);
cmp = c.compareIgnoringPrefix(common, greater.getBuffer(), 
greater.getOffset()
+ KeyValue.ROW_OFFSET, greater.getKeyLength(), less.getBuffer(),
less.getOffset() + KeyValue.ROW_OFFSET, less.getKeyLength());
assertTrue(cmp  0);
  }
{noformat}

Thanks, Lars  Ted.

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch, PerformanceTestCase-6200-94.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-25 Thread Jieshan Bean (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401124#comment-13401124
 ] 

Jieshan Bean commented on HBASE-6200:
-

1,000,000 * 2 comparison, 100ms Vs 121ms. In real environment, this is a 
magnified value.
Since it's a bug, we should discuss whether this decline is tolerable. If not, 
how to optimize and fix this problem? 

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90-v2.patch, 
 HBASE-6200-90.patch, HBASE-6200-92-v2.patch, HBASE-6200-92.patch, 
 HBASE-6200-94-v2.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch, PerformanceTestCase-6200-94.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-24 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400205#comment-13400205
 ] 

Lars Hofhansl commented on HBASE-6200:
--

Also with this change, since KVs are compared extremely frequently we should 
performance test this.


 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90.patch, 
 HBASE-6200-92.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-24 Thread Jieshan Bean (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400248#comment-13400248
]

Jieshan Bean commented on HBASE-6200:
-

bq.Patch looks good. I'm not sure I'm clear what commonPrefix is... how its
used. For example, what is going on here:
bq.We move from commonPrefix+4 to commonPrefix+6. Is this commonPrefix w/i the
row only? Or is it row+? It seems its row? Its not clear from the javadoc nor
as I walk through.

Sorry for the confusion. I will add more comments to explain this:). I will
refresh the patches today.
bq.Probably we should use variable of same name instead of commonPrefix.

Ya...I should use commonLength:)

bq.Also with this change, since KVs are compared extremely frequently we should
performance test this.

Lars, do you have any suggestions on this?:)

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Attachments: 6200-trunk-v2.patch, HBASE-6200-90.patch,
HBASE-6200-92.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch,
HBASE-6200-trunk.patch

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400252#comment-13400252
]

Hadoop QA commented on HBASE-6200:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12533236/HBASE-6200-trunk-v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 11 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2243//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2243//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2243//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2243//console

This message is automatically generated.

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Attachments: 6200-trunk-v2.patch, HBASE-6200-90.patch,
HBASE-6200-92.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch,
HBASE-6200-trunk.patch

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-24 Thread Zhihong Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400254#comment-13400254
 ] 

Zhihong Ted Yu commented on HBASE-6200:
---

For performance test, how about using LoadTestTool / ycsb (same parameters) on 
builds with and without this patch ?

 KeyComparator.compareWithoutRow can be wrong when families have the same 
 prefix
 ---

 Key: HBASE-6200
 URL: https://issues.apache.org/jira/browse/HBASE-6200
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1, 0.94.0
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Blocker
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: 6200-trunk-v2.patch, HBASE-6200-90.patch, 
 HBASE-6200-92.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch, 
 HBASE-6200-trunk.patch


 As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird 
 behavior when some families share the same prefix. He posted a link to his 
 code to show how it fails, http://pastebin.com/7TBA1XGh
 Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families 
 and qualifiers so f:a is said to be bigger than f1:, which is false. Then 
 what happens is that the KVs are returned in the right order from the RS but 
 then doing {{Result.binarySearch}} it uses 
 {{KeyComparator.compareWithoutRow}} which has a different sorting so the end 
 result is undetermined.
 I added some debug and I can see that the data is returned in the right order 
 but {{Arrays.binarySearch}} returned the wrong KV, which is then verified 
 agains the passed family and qualifier which fails so null is returned.
 I don't know how frequent it is for users to have families with the same 
 prefix, but those that do have that and that use those families at the same 
 time will have big correctness issues. This is why I mark this as a blocker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

2012-06-24 Thread Jieshan Bean (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400256#comment-13400256
]

Jieshan Bean commented on HBASE-6200:
-

I agree with this idea. We should create one table with several families which
have the same prefix. And test the load/scan performance.
Before that, I can write one test case to test that method only(w/ and w/o this
patch). And then decide whether do more tests. Is that ok?

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Attachments: 6200-trunk-v2.patch, HBASE-6200-90.patch,
HBASE-6200-92.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch,
HBASE-6200-trunk.patch

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400264#comment-13400264
]

Hadoop QA commented on HBASE-6200:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533237/6200-trunk-v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 11 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2244//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2244//console

This message is automatically generated.

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Attachments: 6200-trunk-v2.patch, HBASE-6200-90.patch,
HBASE-6200-92.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch,
HBASE-6200-trunk.patch

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix

[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400265#comment-13400265
]

Hadoop QA commented on HBASE-6200:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12533238/HBASE-6200-trunk-v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 11 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2245//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2245//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2245//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2245//console

This message is automatically generated.

KeyComparator.compareWithoutRow can be wrong when families have the same
prefix
---

Attachments: 6200-trunk-v2.patch, HBASE-6200-90.patch,
HBASE-6200-92.patch, HBASE-6200-94.patch, HBASE-6200-trunk-v2.patch,
HBASE-6200-trunk.patch

[jira] [Commented] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix