[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356845#comment-15356845
]
Heng Chen commented on HBASE-14279:
---
Thanks for take it up. Please go on. [~ikeda]
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14279-V8-ikeda-branch-1.patch,
> HBASE-14279-V8-ikeda-master.patch, HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356764#comment-15356764
]
Hadoop QA commented on HBASE-14279:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 29s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.regionserver.TestTableBasedReplicationSourceManagerImpl
|
| | hadoop.hbase.client.TestHCM |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12815383/HBASE-14279-V8-ikeda-master.patch
|
| JIRA
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356599#comment-15356599
]
Hiroshi Ikeda commented on HBASE-14279:
---
bq. If someone want it, open another issue for it.
Oops, sorry I overlooked.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14279-V8-ikeda-branch-1.patch,
> HBASE-14279-V8-ikeda-master.patch, HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356521#comment-15356521
]
Hiroshi Ikeda commented on HBASE-14279:
---
I again re-checked and considered how ConcurrentIndex is used, and I realized
that it is enough to just remove ConcurrentIndex at all and use
ConcurrentSkipListSet for BucketCache.blocksByHFile with a
lexicographic comparator. Instead of calling
{{blocksByHFile.values(hfileName)}}, call {{blocksByHFile.subSet(new
BlockCacheKey(hfileName, Long.MIN_VALUE), true, new BlockCacheKey(hfileName,
Long.MAX_VALUE, true))}} without necessity of copying its return value.
FYI, about hashCode, Doug Lea also committed the same logic of the old JDK's
hashCode in his repository. We should be safe even if we accidentally write the
almost same code for such a straight logic.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354872#comment-15354872
]
Heng Chen commented on HBASE-14279:
---
The issue pending here so long time. Let me resolve it as invalid. If someone
want it, open another issue for it.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15072528#comment-15072528
]
Hiroshi Ikeda commented on HBASE-14279:
---
The last patch contains a hash function coming from old JDK, and is it OK
around a license issue? Is there some reason JDK changes the logic of the hash
function and can we ignore it safely? I also slightly worry about there still
exists an other VM implementation that uses the old hash function, making our
internal maps completely waste.
As for other hash calculations, I'm not knowledgeable about MurmurHash but
using a strict calculation might waste. VM might prepare Object.hashCode with
poor dispersion because of performance, and hash functions in hash maps are
just for dispersion in lower bits, which are supposed to be extracted with some
lower bits mask. ConcurrehtHashMap.hash is exceptional because higher bits is
also used to select a segment, and I think it would pay additional performance
cost and it would not be appropriate in our case.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15072574#comment-15072574
]
Heng Chen commented on HBASE-14279:
---
{quote}
The last patch contains a hash function coming from old JDK, and is it OK
around a license issue?
{quote}
org.apache.hadoop.hbase.regionserver.LruHashMap#hashcode already use it, So
IMO it is ok for license.
{quote}
Is there some reason JDK changes the logic of the hash function and can we
ignore it safely? I also slightly worry about there still exists an other VM
implementation that uses the old hash function, making our internal maps
completely waste.
{quote}
I don't think there is any problem here. As i mentioned above, if orthogonal
problem exists, the only cost is some conflicts when search key in internal
map. And i do NOT think it is a big problem compared with disk IO. And if we
check the usage of ConcurrentIndex, there is no GET operation. So i think we
have no need to do improvement for internal map operation, it is useless for
the whole performance of BucketCache, wdyt? [~ikeda]
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15072384#comment-15072384
]
Heng Chen commented on HBASE-14279:
---
{quote}
I worry about whether the hash calculation is well orthogonal with
HashMap.hash() so that objects are well-distributed among the entries in each
internal map after the objects are distributed into the internal maps. The new
calculation seems coming from JDK 1.4.
{quote}
You are right. The hash calculation from HashMap maybe has orthogonal problem.
After one Key dispatch into one map, the cost is some conflicts when search
the key in map, but i think it is OK now compared with disk IO. Even IMO we
could do hash for key with MOD operations only. I don't think it is a big
problem. wdyt? [~ikeda]
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070375#comment-15070375
]
Hiroshi Ikeda commented on HBASE-14279:
---
I worry about whether the hash calculation is well orthogonal with
HashMap.hash() so that objects are well-distributed among the entries in each
internal map after the objects are distributed into the internal maps. The new
calculation seems coming from JDK 1.4.
As for the variant of single-word Wang/Jenkins hash, in
http://gee.cs.oswego.edu/dl/concurrency-interest/index.html
{quote}
Sources for all classes originated by the JSR166 group are released to the
public domain, as described at
http://creativecommons.org/licenses/publicdomain. This includes all code in
java.util.concurrent and its subpackages (except CopyOnWriteArrayList), ...
{quote}
and Doug Lea introduced the hash calculation code at the revision 1.93 in his
repository:
http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/main/java/util/concurrent/ConcurrentHashMap.java?revision=1.93
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070496#comment-15070496
]
Hiroshi Ikeda commented on HBASE-14279:
---
I misunderstood for some reason and the same thing can be said for the variant
of single-word Wang/Jenkins hash. That's deep-rooted :(
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070502#comment-15070502
]
Hadoop QA commented on HBASE-14279:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779334/HBASE-14279_v7.1.patch
against master branch at commit 8e0854c64be553595b8ed44b9856a3d74ad3005f.
ATTACHMENT ID: 12779334
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorEndpoint
org.apache.hadoop.hbase.master.TestMaster
org.apache.hadoop.hbase.master.procedure.TestDeleteNamespaceProcedure
org.apache.hadoop.hbase.util.TestHBaseFsckReplicas
org.apache.hadoop.hbase.master.procedure.TestCreateNamespaceProcedure
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.master.TestHMasterRPCException
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.master.procedure.TestModifyNamespaceProcedure
org.apache.hadoop.hbase.client.TestMobSnapshotFromClient
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
org.apache.hadoop.hbase.client.TestSnapshotFromClient
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient
org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove
org.apache.hadoop.hbase.client.TestAdmin1
org.apache.hadoop.hbase.master.handler.TestTableDeleteFamilyHandler
org.apache.hadoop.hbase.regionserver.TestMobStoreScanner
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClientWithRegionReplicas
org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient
org.apache.hadoop.hbase.client.TestLeaseRenewal
org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas
org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile
org.apache.hadoop.hbase.quotas.TestQuotaThrottle
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17002//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17002//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17002//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17002//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.1.patch,
> HBASE-14279_v7.patch, LockStripedBag.java
>
>
>
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068374#comment-15068374
]
stack commented on HBASE-14279:
---
What [~ikeda] says... if really copied, thats a problem (CHM is not like the
other Doug Lea stuff which is in public domain). There are a bunch of write ups
on this hash in particular. Just do a murmur hash on it? Or copy the wikipedia
page (though seems like there have been improvements done in CHM).
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069350#comment-15069350
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779189/HBASE-14279_v7.patch
against master branch at commit 1af98f255132ef6716a1f6ba1d8d71a36ea38840.
ATTACHMENT ID: 12779189
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16989//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16989//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16989//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16989//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, HBASE-14279_v7.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067585#comment-15067585
]
Hiroshi Ikeda commented on HBASE-14279:
---
I have second thought about the sentence:
{quote}
+ // Copied from ConcurrentHashMap
{quote}
It is better to remove the sentence because of avoiding a license issue. I hope
that the logic of the variant of single-word Wang/Jenkins hash is well-known
ordinary one.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067510#comment-15067510
]
Heng Chen commented on HBASE-14279:
---
I need one more +1 before commit.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066238#comment-15066238
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778788/HBASE-14279_v5.patch
against master branch at commit e75e26e3c6cfe7fd378081839d60fc711c1e095f.
ATTACHMENT ID: 12778788
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16947//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16947//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16947//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16947//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066325#comment-15066325
]
Heng Chen commented on HBASE-14279:
---
[~ikeda] Thanks for your good suggestions. I update patch.
I use TreeSet instead of ConcurrentSkipListSet and remove
DefaultValueSetFactory directly.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067672#comment-15067672
]
Heng Chen commented on HBASE-14279:
---
{quote}
It is better to remove the sentence because of avoiding a license issue.
{quote}
OK, i will fix it when commit. Any suggestions? [~stack]
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066425#comment-15066425
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778809/HBASE-14279_v6.patch
against master branch at commit e75e26e3c6cfe7fd378081839d60fc711c1e095f.
ATTACHMENT ID: 12778809
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16950//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16950//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16950//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16950//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, HBASE-14279_v6.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066143#comment-15066143
]
Hiroshi Ikeda commented on HBASE-14279:
---
{quote}
Using a concurrent or synchronized set wastes completely, because their
instances are always accessed under locking segments.
{quote}
In addition, using a static nested class spoils some visibility restriction.
There is no reason to use a package private class.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> HBASE-14279_v5.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998256#comment-14998256
]
Hiroshi Ikeda commented on HBASE-14279:
---
Sorry, these days ConcurrentHashMap seems much more sophisticated using
UNSAFE...
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998240#comment-14998240
]
Hiroshi Ikeda commented on HBASE-14279:
---
To tell the truth ConcurrentHashMap is more sophisticated and has advantage to
a simple striped map; ConcurrentHashMap.get just has a momentary critical path,
and it is a well-known idiom to try get before putIfAbsent. I'm not sure how to
implement the similar feature in this issue.
{code}
+private static class DefaultValueSetFactory implements Supplier
{
...skip...
+ public Set get() {
+return new ConcurrentSkipListSet(comparator);
+ }
{code}
Using a concurrent or synchronized set wastes completely, because their
instances are always accessed under locking segments.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998353#comment-14998353
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771510/HBASE-14279_v5.patch
against master branch at commit 112900d0425a8157b89041f0e353ebf5cc259c69.
ATTACHMENT ID: 12771510
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16475//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16475//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16475//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16475//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16475//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, HBASE-14279_v5.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996970#comment-14996970
]
stack commented on HBASE-14279:
---
bq. We really need one DataStructure 'multiMapSet'.
Where else we need this [~chenheng]? KeyLocker?
I like the suggestion of taking the class out of util and down into bucketcache
package and making it package private ensuring it works well for the BC case.
We can then come back and tackle making it general later?
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998045#comment-14998045
]
Heng Chen commented on HBASE-14279:
---
{quote}
Where else we need this Heng Chen? KeyLocker?
{quote}
yeah, currently IdLock, KeyLocker NOT use MultiMap inside.
IMO if concurrent conflicts is not heavy, may be we can use MultiMap to reduce
memory used.
Of course, it has no relates with this issue. we can come back when we really
need to do it.
Thanks for your suggestion [~ikeda] and [~stack].
I will move ConcurrentIndex into BucketCache.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996278#comment-14996278
]
Heng Chen commented on HBASE-14279:
---
Thanks [~ikeda]. Let's see what the other talk about. [~stack] :)
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14995958#comment-14995958
]
Hiroshi Ikeda commented on HBASE-14279:
---
I'm not sure whether you are creating a class for general usage or for a
specific usage. You might pick some of their merits if you make more effort to
abstract the class, though it wouldn't pay.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992097#comment-14992097
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770824/HBASE-14279_v3.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770824
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16410//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993240#comment-14993240
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770950/HBASE-14279_v4.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12770950
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1728 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992856#comment-14992856
]
Hiroshi Ikeda commented on HBASE-14279:
---
{code}
public Set values(K key) {
-return container.get(key);
+Map seg = segments(key);
+return seg.get(key);
}
{code}
Defensively copy under the lock of the segment (the reason was already
commented a few month ago), and consequently using valueSetFactory is almost
meaningless:
{code}
+ Set set = seg.get(key);
+ if (set == null) {
+set = valueSetFactory.get();
+seg.put(key, set);
}
{code}
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993063#comment-14993063
]
Heng Chen commented on HBASE-14279:
---
{quote}
Defensively copy under the lock of the segment
{quote}
Make sense, fix it.
{quote}
valueSetFactory is almost meaningless
{quote}
I don't think so, because BucketCache will define sortedSet by itself, so we
should pass the comparator into it.
I add some tests to verify this class.
Thanks [~ikeda] a lot. update patch
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993302#comment-14993302
]
Hiroshi Ikeda commented on HBASE-14279:
---
{quote}
Currently there is only one place to call this method, it is is BucketCache.
{quote}
I think ConcurrentIndex is never appropriate for the package hbase.util. This
is a part of BucketCahce and should be placed in the same package of
BucketCahce, and it is better to be a package private class. (Fortunately this
is still declared as InterfaceAudience.Private.)
Anyone would not so much complain encapsulated classes' specification as long
as they well work together.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993313#comment-14993313
]
Heng Chen commented on HBASE-14279:
---
{quote}
I think ConcurrentIndex is never appropriate for the package hbase.util.
{quote}
I don't think so. We really need one DataStructure 'multiMapSet'. And
concurrentIndex can do it, maybe we could add more interface for it. wdyt? :)
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993241#comment-14993241
]
Hiroshi Ikeda commented on HBASE-14279:
---
{quote}
BucketCache will define sortedSet by itself, so we should pass the comparator
into it.
{quote}
Oops, I overlooked... Is it important to be able to select between hash-set and
sorted-set? Anyway concurrency/synchronized one just gives nothing but
overhead, and at least it is required to rewrite the default factory.
{code}
public Set values(K key) {
...skip...
+ return ImmutableSet.copyOf(set);
+}
}
{code}
I think the caller would expect the returned object is created by the given
value factory, and some methods might have different behaviors from the
immutable set, especially if the caller gives a value factory which creates a
sorted set (or a value comparator).
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993268#comment-14993268
]
Heng Chen commented on HBASE-14279:
---
{quote}
Is it important to be able to select between hash-set and sorted-set?
{quote}
Yeah, i think we need to do it as original logic.
{quote}
I think the caller would expect the returned object is created by the given
value factory, and some methods might have different behaviors from the
immutable set, especially if the caller gives a value factory which creates a
sorted set (or a value comparator).
{quote}
Currently there is only one place to call this method, it is is BucketCache.
After call it, it use ImmutableSet.copyOf(set) to make another set to read.
IMO we can return this immutable set directly by ConcurrentIndex.values.
wdyt? :)
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988401#comment-14988401
]
stack commented on HBASE-14279:
---
Can we get the LockedStripedBag into a state where it could be tried in place
of ConcurrentIndex?
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988862#comment-14988862
]
Hiroshi Ikeda commented on HBASE-14279:
---
I'm ashamed that I used the word "bag" because I just wanted to use a different
word than "multimap", but the Google collections provides the interface
"SetMultimap", which might be an appropriate name.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976171#comment-14976171
]
Heng Chen commented on HBASE-14279:
---
Sorry for being late, i think we should see what the others talk about?
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944386#comment-14944386
]
Hiroshi Ikeda commented on HBASE-14279:
---
You cannot freely use the internal striped locks in the ConcurrentHashMap and
you have proved that sticking to ConcurrentHashMap results in the surprisingly
complex and overheaded logic. Also there is still a problem of breaking the
invariant, as mentioned above.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944268#comment-14944268
]
Heng Chen commented on HBASE-14279:
---
{quote}
Added an example of lock striping.
{quote}
Why not use {{ConcurrentHashMap}}, it is realized of lock striping
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14942891#comment-14942891
]
Hiroshi Ikeda commented on HBASE-14279:
---
Oops, {{remove}} contains a bug :(
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723055#comment-14723055
]
Heng Chen commented on HBASE-14279:
---
{quote}
There is a similar class, KeyLoccker, whose key type is a generic one, and you
can use it without copying its implementation logic. The implementation of
IdLocker and KeyLocker is not good and should be re-written, and I believe it
is better to avoid to copy their implementation.
{quote}
yeah, you are right. {{KeyLocker}} is a better choice.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723201#comment-14723201
]
Heng Chen commented on HBASE-14279:
---
It has a problem.
{code}
public class KeyLocker> {
{code}
KeyLocker require the the {{K}} is {{Comparable}}, But key in
{{ConcurrentIndex}} has no constraint.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724726#comment-14724726
]
Hiroshi Ikeda commented on HBASE-14279:
---
Sorry I didn't realize {{KeyLocker}} has such restriction. {{KeyLocker}} should
not have such restriction.
Anyway, it is better to use lock striping rather than such lock classes.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718316#comment-14718316
]
Hiroshi Ikeda commented on HBASE-14279:
---
There is a similar class, {{KeyLoccker}}, whose key type is a generic one, and
you can use it without copying its implementation logic. The implementation of
{{IdLocker}} and {{KeyLocker}} is not good and should be re-written, and I
believe it is better to avoid to copy their implementation.
To begin with, using 2 objects of {{ConcurrentHashMap}} is not good because
{{putIfAbset}} and {{remove}} use just lock striping internally, and the
periods of time which may cause conflict between threads increases twofold. It
is better to manually use lock striping instead of {{ConcurrentHashMap}}.
BTW, {{ConcurrentHashMap.get}} just locks the segment only when accessing a
value, and it is an idem to use {{get}} before {{putIfAbsent}}. This has
potentially to reduce granularity of locks, though it seems not trivial to
implement.
There is another thing to concern about. From the point of the view of
object-oriented programing, exposing internal values by the method {{values}}
causes breaking the invariant that the empty set should be removed from the
map. It should be defensively copied, though it is required to change the API,
and it may also be required to change code of the clients which use this class,
making the patch easier to become stale.
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Assignee: Heng Chen
Priority: Minor
Attachments: HBASE-14279.patch, HBASE-14279_v2.patch
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718440#comment-14718440
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12752702/HBASE-14279_v2.patch
against master branch at commit cc1542828de93b8d54cc14497fd5937989ea1b6d.
ATTACHMENT ID: 12752702
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1850 checkstyle errors (more than the master's current 1849 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 10 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatTestBase.testScan(MultiTableInputFormatTestBase.java:255)
at
org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatTestBase.testScanEmptyToAPP(MultiTableInputFormatTestBase.java:202)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleHFileSplit(TestLoadIncrementalHFiles.java:153)
at
org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterExclusiveColumn(TestRowCounter.java:111)
at
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.testMROnTableWithDeletes(TestImportTSVWithVisibilityLabels.java:182)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testRegionCrossingHFileSplit(TestLoadIncrementalHFiles.java:193)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testRegionCrossingHFileSplit(TestLoadIncrementalHFiles.java:171)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleHFileSplit(TestLoadIncrementalHFiles.java:153)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15315//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15315//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15315//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15315//console
This message is automatically generated.
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Assignee: Heng Chen
Priority: Minor
Attachments: HBASE-14279.patch, HBASE-14279_v2.patch
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716135#comment-14716135
]
Hiroshi Ikeda commented on HBASE-14279:
---
{code}
public void put(K key, V value) {
...
PairObject, SetV existing = container.putIfAbsent(key, lockAndSet);
if (existing != null) {
synchronized (existing.getFirst()) {
existing = container.get(key); //Re check if the key is associate with
the set.
...
{code}
I think you recognize that, the element given from {{container.putIfAbset}}
might be removed the next moment, so that you lock and re-check. But in that
case you just lock the obsolete object that is already removed from the
container, and you cannot prevent the other overtaking threads from replacing
the element.
Even if you would again lock the object inside the re-checked element, the
rechecked element may be also removed from the container the next moment for
the same reason, and the code should be as follows:
{code}
public void put(K key, V value) {
...
PairObject, SetV existing = container.putIfAbsent(key, lockAndSet);
if (existing != null) {
synchronized (existing.getFirst()) {
existing = container.putIfAbsent(key, lockAndSet);
if (existing != null) {
synchronized (existing.getFirst()) {
existing = container.putIfAbsent(key, lockAndSet);
...
(It continues to infinity.)
{code}
After all, you cannot atomically add/remove elements with some additional
processing under locking objects inside the container.
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Assignee: Heng Chen
Priority: Minor
Attachments: HBASE-14279.patch
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716198#comment-14716198
]
Heng Chen commented on HBASE-14279:
---
{quote}
But in that case you just lock the obsolete object that is already removed
from the container, and you cannot prevent the other overtaking threads from
replacing the element.
{quote}
I see I think we can use something like {{IdLock}} to lock {{Set}}
associate with some one key.
I will update the patch.
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Assignee: Heng Chen
Priority: Minor
Attachments: HBASE-14279.patch
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710787#comment-14710787
]
Hiroshi Ikeda commented on HBASE-14279:
---
Go ahead :-)
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Assignee: Heng Chen
Priority: Minor
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710691#comment-14710691
]
Heng Chen commented on HBASE-14279:
---
May I upload a patch ?
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Priority: Minor
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712256#comment-14712256
]
Hiroshi Ikeda commented on HBASE-14279:
---
Exclusive access is required to lock the same object with threads, and you
cannot prevent from adding/removing a element to/from the concurrent
collections with locking an object which may be already replaced after you get
it.
Sorry I have no time just now, and details later.
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Assignee: Heng Chen
Priority: Minor
Attachments: HBASE-14279.patch
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709066#comment-14709066
]
Hiroshi Ikeda commented on HBASE-14279:
---
It is better to use lock striping because the type of internal sets are
arbitrary and it is not possible to just take a snapshot by the {{values}}
method, or change the specification at all.
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Priority: Minor
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706430#comment-14706430
]
Heng Chen commented on HBASE-14279:
---
yeah... maybe we can replace {{ConcurrentMapK, SetV container;}} with
{{ConcurrentMapK, PairObject, SetV container;}}
{{Pair.getFirst()}} is Lock for current key. {{Pair.getSecond()}} is real set.
Race condition in ConcurrentIndex
-
Key: HBASE-14279
URL: https://issues.apache.org/jira/browse/HBASE-14279
Project: HBase
Issue Type: Bug
Reporter: Hiroshi Ikeda
Priority: Minor
{{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
to remove a non-empty set, and to add a value to a removed set. Also
{{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
