[jira] [Commented] (HBASE-16992) The usage of mutation from CP is weird.
[
https://issues.apache.org/jira/browse/HBASE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635358#comment-15635358
]
Anoop Sam John commented on HBASE-16992:
Ya #1 is ok. Pls add some comments.
> The usage of mutation from CP is weird.
> ---
>
> Key: HBASE-16992
> URL: https://issues.apache.org/jira/browse/HBASE-16992
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
>
> {code:title=HRegion#doMiniBatchMutate|borderStyle=solid}
> Mutation cpMutation = cpMutations[j];
> Map cpFamilyMap = cpMutation.getFamilyCellMap();
> checkAndPrepareMutation(cpMutation, replay, cpFamilyMap, now);
> // Acquire row locks. If not, the whole batch will fail.
> acquiredRowLocks.add(getRowLockInternal(cpMutation.getRow(), true));
> if (cpMutation.getDurability() == Durability.SKIP_WAL) {
> recordMutationWithoutWal(cpFamilyMap);
> }
> // Returned mutations from coprocessor correspond to the Mutation at index i.
> We can
> // directly add the cells from those mutations to the familyMaps of this
> mutation.
> mergeFamilyMaps(familyMaps[i], cpFamilyMap); // will get added to the
> memstore later
> {code}
> 1. Does the returned mutation from coprocessor have the same row as the
> corresponded mutation? If so, the acquiredRowLocks() can be saved. If not,
> the corresponded mutation may maintain the cells with different row due to
> mergeFamilyMaps().
> 2. Is returned mutation's durability useful? If so, we should deal with the
> different durabilities before mergeFamilyMaps(). If not, the
> recordMutationWithoutWal can be saved.
> 3. If both the returned mutation and corresponded mutation have
> Durability.SKIP_WAL, the recordMutationWithoutWal() may record the duplicate
> cells due to mergeFamilyMaps().
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17006) Add names to threads for better debugability of thread dumps
[ https://issues.apache.org/jira/browse/HBASE-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635349#comment-15635349 ] Hudson commented on HBASE-17006: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #67 (See [https://builds.apache.org/job/HBase-1.3-JDK8/67/]) HBASE-17006 Give name to existing threads. Having thread names in logs (appy: rev 3dc95c0e679ad6b54ecfbce1d35eec2a99868e10) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/monkies/PolicyBasedChaosMonkey.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriterBase.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/JvmPauseMonitor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluationCommons.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java > Add names to threads for better debugability of thread dumps > > > Key: HBASE-17006 > URL: https://issues.apache.org/jira/browse/HBASE-17006 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-17006.master.001.patch, > HBASE-17006.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16570) Compute region locality in parallel at startup
[
https://issues.apache.org/jira/browse/HBASE-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635352#comment-15635352
]
Yu Li commented on HBASE-16570:
---
>From our online usage we found at master startup the thread number might blow
>up simply because we're not limiting the maximum thread number of the executor
>used in {{RegionLocationFinder}}:
{code}
RegionLocationFinder() {
this.cache = createCache();
executor = MoreExecutors.listeningDecorator(
Executors.newScheduledThreadPool(
5,
new ThreadFactoryBuilder().
setDaemon(true)
.setNameFormat("region-location-%d")
.build()));
}
{code}
And since in patch here we will submit as many as region number tasks to get
block locations, we should add some limit on thread number in parallel as well
as preventing all threads starting at the same time (like use executor.schedule
instead of executor.submit).
[~aoxiang] please upload fix of this issue as another addendum, thanks.
> Compute region locality in parallel at startup
> --
>
> Key: HBASE-16570
> URL: https://issues.apache.org/jira/browse/HBASE-16570
> Project: HBase
> Issue Type: Sub-task
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
> Attachments: HBASE-16570-master_V1.patch,
> HBASE-16570-master_V2.patch, HBASE-16570-master_V3.patch,
> HBASE-16570-master_V4.patch, HBASE-16570.branch-1.3-addendum.patch,
> HBASE-16570_addnum.patch, HBASE-16570_addnum_v2.patch,
> HBASE-16570_addnum_v3.patch, HBASE-16570_addnum_v4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16992) The usage of mutation from CP is weird.
[
https://issues.apache.org/jira/browse/HBASE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635350#comment-15635350
]
ChiaPing Tsai commented on HBASE-16992:
---
bq. Pls note that the merge method merges the cell sets into a single set.
There is no Mutation change happening here. This collection of cells is what
treated as a single unit of write. So IMO there is no breaking of the Row
agreement. It should be ok.
I get it. Thanks for your explanation.
bq. Also as said, we dont consider Durability from CP. We will go with same
Durability as that of the corresponding Mutations in batch. Any checks and fail
needed? May be not. Pls add some comments around code that we do this way.
There are two solutions.
# Add some comments, remove the recordMutationWithoutWal() when merging, and
just ignore the durability from CP.
# Checks and fail if the durability is not matched.
The first solution is the eventual decision? If so, i will attach the patch
asap.
> The usage of mutation from CP is weird.
> ---
>
> Key: HBASE-16992
> URL: https://issues.apache.org/jira/browse/HBASE-16992
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
>
> {code:title=HRegion#doMiniBatchMutate|borderStyle=solid}
> Mutation cpMutation = cpMutations[j];
> Map cpFamilyMap = cpMutation.getFamilyCellMap();
> checkAndPrepareMutation(cpMutation, replay, cpFamilyMap, now);
> // Acquire row locks. If not, the whole batch will fail.
> acquiredRowLocks.add(getRowLockInternal(cpMutation.getRow(), true));
> if (cpMutation.getDurability() == Durability.SKIP_WAL) {
> recordMutationWithoutWal(cpFamilyMap);
> }
> // Returned mutations from coprocessor correspond to the Mutation at index i.
> We can
> // directly add the cells from those mutations to the familyMaps of this
> mutation.
> mergeFamilyMaps(familyMaps[i], cpFamilyMap); // will get added to the
> memstore later
> {code}
> 1. Does the returned mutation from coprocessor have the same row as the
> corresponded mutation? If so, the acquiredRowLocks() can be saved. If not,
> the corresponded mutation may maintain the cells with different row due to
> mergeFamilyMaps().
> 2. Is returned mutation's durability useful? If so, we should deal with the
> different durabilities before mergeFamilyMaps(). If not, the
> recordMutationWithoutWal can be saved.
> 3. If both the returned mutation and corresponded mutation have
> Durability.SKIP_WAL, the recordMutationWithoutWal() may record the duplicate
> cells due to mergeFamilyMaps().
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17006) Add names to threads for better debugability of thread dumps
[ https://issues.apache.org/jira/browse/HBASE-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635310#comment-15635310 ] Hudson commented on HBASE-17006: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #55 (See [https://builds.apache.org/job/HBase-1.2-JDK8/55/]) HBASE-17006 Give name to existing threads. Having thread names in logs (appy: rev 326ecc32fd4b9f37e8a77a9b592d301f38eece13) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriterBase.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluationCommons.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/monkies/PolicyBasedChaosMonkey.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/JvmPauseMonitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java > Add names to threads for better debugability of thread dumps > > > Key: HBASE-17006 > URL: https://issues.apache.org/jira/browse/HBASE-17006 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-17006.master.001.patch, > HBASE-17006.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635307#comment-15635307 ] Anoop Sam John commented on HBASE-16630: We would like to enable BucketCache On by default for the data blocks in 2.0. So can we up the priority to be critical? [~dvdreddy] u still working on this? > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630-v2.patch, HBASE-16630-v3.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16992) The usage of mutation from CP is weird.
[
https://issues.apache.org/jira/browse/HBASE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16992:
---
Fix Version/s: 1.3.1
1.4.0
2.0.0
> The usage of mutation from CP is weird.
> ---
>
> Key: HBASE-16992
> URL: https://issues.apache.org/jira/browse/HBASE-16992
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
>
> {code:title=HRegion#doMiniBatchMutate|borderStyle=solid}
> Mutation cpMutation = cpMutations[j];
> Map cpFamilyMap = cpMutation.getFamilyCellMap();
> checkAndPrepareMutation(cpMutation, replay, cpFamilyMap, now);
> // Acquire row locks. If not, the whole batch will fail.
> acquiredRowLocks.add(getRowLockInternal(cpMutation.getRow(), true));
> if (cpMutation.getDurability() == Durability.SKIP_WAL) {
> recordMutationWithoutWal(cpFamilyMap);
> }
> // Returned mutations from coprocessor correspond to the Mutation at index i.
> We can
> // directly add the cells from those mutations to the familyMaps of this
> mutation.
> mergeFamilyMaps(familyMaps[i], cpFamilyMap); // will get added to the
> memstore later
> {code}
> 1. Does the returned mutation from coprocessor have the same row as the
> corresponded mutation? If so, the acquiredRowLocks() can be saved. If not,
> the corresponded mutation may maintain the cells with different row due to
> mergeFamilyMaps().
> 2. Is returned mutation's durability useful? If so, we should deal with the
> different durabilities before mergeFamilyMaps(). If not, the
> recordMutationWithoutWal can be saved.
> 3. If both the returned mutation and corresponded mutation have
> Durability.SKIP_WAL, the recordMutationWithoutWal() may record the duplicate
> cells due to mergeFamilyMaps().
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16992) The usage of mutation from CP is weird.
[
https://issues.apache.org/jira/browse/HBASE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635303#comment-15635303
]
Anoop Sam John commented on HBASE-16992:
You will give a patch? :-)
> The usage of mutation from CP is weird.
> ---
>
> Key: HBASE-16992
> URL: https://issues.apache.org/jira/browse/HBASE-16992
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
>
> {code:title=HRegion#doMiniBatchMutate|borderStyle=solid}
> Mutation cpMutation = cpMutations[j];
> Map cpFamilyMap = cpMutation.getFamilyCellMap();
> checkAndPrepareMutation(cpMutation, replay, cpFamilyMap, now);
> // Acquire row locks. If not, the whole batch will fail.
> acquiredRowLocks.add(getRowLockInternal(cpMutation.getRow(), true));
> if (cpMutation.getDurability() == Durability.SKIP_WAL) {
> recordMutationWithoutWal(cpFamilyMap);
> }
> // Returned mutations from coprocessor correspond to the Mutation at index i.
> We can
> // directly add the cells from those mutations to the familyMaps of this
> mutation.
> mergeFamilyMaps(familyMaps[i], cpFamilyMap); // will get added to the
> memstore later
> {code}
> 1. Does the returned mutation from coprocessor have the same row as the
> corresponded mutation? If so, the acquiredRowLocks() can be saved. If not,
> the corresponded mutation may maintain the cells with different row due to
> mergeFamilyMaps().
> 2. Is returned mutation's durability useful? If so, we should deal with the
> different durabilities before mergeFamilyMaps(). If not, the
> recordMutationWithoutWal can be saved.
> 3. If both the returned mutation and corresponded mutation have
> Durability.SKIP_WAL, the recordMutationWithoutWal() may record the duplicate
> cells due to mergeFamilyMaps().
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16992) The usage of mutation from CP is weird.
[
https://issues.apache.org/jira/browse/HBASE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16992:
---
Affects Version/s: 1.4.0
1.3.0
2.0.0
> The usage of mutation from CP is weird.
> ---
>
> Key: HBASE-16992
> URL: https://issues.apache.org/jira/browse/HBASE-16992
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
>
> {code:title=HRegion#doMiniBatchMutate|borderStyle=solid}
> Mutation cpMutation = cpMutations[j];
> Map cpFamilyMap = cpMutation.getFamilyCellMap();
> checkAndPrepareMutation(cpMutation, replay, cpFamilyMap, now);
> // Acquire row locks. If not, the whole batch will fail.
> acquiredRowLocks.add(getRowLockInternal(cpMutation.getRow(), true));
> if (cpMutation.getDurability() == Durability.SKIP_WAL) {
> recordMutationWithoutWal(cpFamilyMap);
> }
> // Returned mutations from coprocessor correspond to the Mutation at index i.
> We can
> // directly add the cells from those mutations to the familyMaps of this
> mutation.
> mergeFamilyMaps(familyMaps[i], cpFamilyMap); // will get added to the
> memstore later
> {code}
> 1. Does the returned mutation from coprocessor have the same row as the
> corresponded mutation? If so, the acquiredRowLocks() can be saved. If not,
> the corresponded mutation may maintain the cells with different row due to
> mergeFamilyMaps().
> 2. Is returned mutation's durability useful? If so, we should deal with the
> different durabilities before mergeFamilyMaps(). If not, the
> recordMutationWithoutWal can be saved.
> 3. If both the returned mutation and corresponded mutation have
> Durability.SKIP_WAL, the recordMutationWithoutWal() may record the duplicate
> cells due to mergeFamilyMaps().
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16992) The usage of mutation from CP is weird.
[
https://issues.apache.org/jira/browse/HBASE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635299#comment-15635299
]
Anoop Sam John commented on HBASE-16992:
bq.The main point i concern is the cells in one mutation may have different
rows, because we merge the cells returned by CP into original mutation.
Pls note that the merge method merges the cell sets into a single set. There is
no Mutation change happening here. This collection of cells is what treated as
a single unit of write. So IMO there is no breaking of the Row agreement. It
should be ok.
So the issue to be handled here is to avoid the duplicate call to
recordMutationWithoutWal (?) We need to avoid the call before merge op.
Also as said, we dont consider Durability from CP. We will go with same
Durability as that of the corresponding Mutations in batch. Any checks and
fail needed? May be not. Pls add some comments around code that we do this way.
> The usage of mutation from CP is weird.
> ---
>
> Key: HBASE-16992
> URL: https://issues.apache.org/jira/browse/HBASE-16992
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>
> {code:title=HRegion#doMiniBatchMutate|borderStyle=solid}
> Mutation cpMutation = cpMutations[j];
> Map cpFamilyMap = cpMutation.getFamilyCellMap();
> checkAndPrepareMutation(cpMutation, replay, cpFamilyMap, now);
> // Acquire row locks. If not, the whole batch will fail.
> acquiredRowLocks.add(getRowLockInternal(cpMutation.getRow(), true));
> if (cpMutation.getDurability() == Durability.SKIP_WAL) {
> recordMutationWithoutWal(cpFamilyMap);
> }
> // Returned mutations from coprocessor correspond to the Mutation at index i.
> We can
> // directly add the cells from those mutations to the familyMaps of this
> mutation.
> mergeFamilyMaps(familyMaps[i], cpFamilyMap); // will get added to the
> memstore later
> {code}
> 1. Does the returned mutation from coprocessor have the same row as the
> corresponded mutation? If so, the acquiredRowLocks() can be saved. If not,
> the corresponded mutation may maintain the cells with different row due to
> mergeFamilyMaps().
> 2. Is returned mutation's durability useful? If so, we should deal with the
> different durabilities before mergeFamilyMaps(). If not, the
> recordMutationWithoutWal can be saved.
> 3. If both the returned mutation and corresponded mutation have
> Durability.SKIP_WAL, the recordMutationWithoutWal() may record the duplicate
> cells due to mergeFamilyMaps().
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16992) The usage of mutation from CP is weird.
[
https://issues.apache.org/jira/browse/HBASE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16992:
---
Priority: Minor (was: Major)
> The usage of mutation from CP is weird.
> ---
>
> Key: HBASE-16992
> URL: https://issues.apache.org/jira/browse/HBASE-16992
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
>
> {code:title=HRegion#doMiniBatchMutate|borderStyle=solid}
> Mutation cpMutation = cpMutations[j];
> Map cpFamilyMap = cpMutation.getFamilyCellMap();
> checkAndPrepareMutation(cpMutation, replay, cpFamilyMap, now);
> // Acquire row locks. If not, the whole batch will fail.
> acquiredRowLocks.add(getRowLockInternal(cpMutation.getRow(), true));
> if (cpMutation.getDurability() == Durability.SKIP_WAL) {
> recordMutationWithoutWal(cpFamilyMap);
> }
> // Returned mutations from coprocessor correspond to the Mutation at index i.
> We can
> // directly add the cells from those mutations to the familyMaps of this
> mutation.
> mergeFamilyMaps(familyMaps[i], cpFamilyMap); // will get added to the
> memstore later
> {code}
> 1. Does the returned mutation from coprocessor have the same row as the
> corresponded mutation? If so, the acquiredRowLocks() can be saved. If not,
> the corresponded mutation may maintain the cells with different row due to
> mergeFamilyMaps().
> 2. Is returned mutation's durability useful? If so, we should deal with the
> different durabilities before mergeFamilyMaps(). If not, the
> recordMutationWithoutWal can be saved.
> 3. If both the returned mutation and corresponded mutation have
> Durability.SKIP_WAL, the recordMutationWithoutWal() may record the duplicate
> cells due to mergeFamilyMaps().
> Any comment? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635287#comment-15635287 ] Anoop Sam John commented on HBASE-17018: If an MR way is what we take, then better it can be an ImportTSV kind of job which splits data into HFiles and do bulk load? > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17013) Add constructor to RowMutations for initializing the capacity of internal list
[ https://issues.apache.org/jira/browse/HBASE-17013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17013: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Timed out tests are knows failures. Pushed to master. Thanks for the patch. > Add constructor to RowMutations for initializing the capacity of internal list > -- > > Key: HBASE-17013 > URL: https://issues.apache.org/jira/browse/HBASE-17013 > Project: HBase > Issue Type: Improvement >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17013.v0.patch > > > The number of mutations is known when converting all ClientProtos.Action to a > single RowMutations. > We can reduce memory consumption by the initial capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen resolved HBASE-16993.
-
Resolution: Not A Bug
caused by hbase.bucketcache.bucket.sizes incorrect value
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1935)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)
> at
[jira] [Commented] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635228#comment-15635228
]
liubangchen commented on HBASE-16993:
-
confirm this exception caused by by property
hbase.bucketcache.bucket.sizes，when i set this property to default then server
work ok,my setting is:
16384,32768,40960, 46000,49152,51200,65536,131072,524288
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1935)
> at
>
[jira] [Comment Edited] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632273#comment-15632273
]
liubangchen edited comment on HBASE-16993 at 11/4/16 4:39 AM:
--
I test all data_block_encoding ,the phenomenon is kile this
DATA_BLOCK_ENCODING load run restart
DIFF | Must happen | Must happen |Must happen |
FAST_DIFF| Must happen | Must happen |Must happen |
PREFIX | sometime| sometime|not happen in test|
PREFIX_TREE | Must happen | Must happen |Must happen |
NONE | sometime| sometime | sometime |
test way is
1. drop table and create table
2. load data
3. run
4. restart server
load 100% write
run 90% read 10% wrtie
was (Author: liubangchen):
I test all data_block_encoding ,the phenomenon is kile this
DATA_BLOCK_ENCODING load run restart
DIFF | Must happen | Must happen |Must happen |
FAST_DIFF| Must happen | Must happen |Must happen |
PREFIX | sometime| sometime|not happen in test|
PREFIX_TREE | Must happen | Must happen |Must happen |
NONE | sometime| sometime | sometime |
test way is
1. drop table and create table
2. load data
3. run
4. restart server
load 100% write
run 90% read 10% wrtie
i think this problem may be closely related to DATA_BLOCK_ENCODING
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
>
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635154#comment-15635154 ] Phil Yang commented on HBASE-17018: --- An interesting and useful feature :) Cassandra has a feature/api called "atomic batches". If the client get response from server, we can say all mutations have be executed. If the request timeout, server can guarantee "all or nothing" -- all will be executed eventually or nothing will be executed. I think we can provide a similar feature to HBase users as a Table's API. And BufferedMutator can have an optional logic to switch to this API when the initial request failed. For implementation, I think we'd better not use MR because users may not have MR for a HBase cluster, or at least RS should be able to replay mutations. If we save mutations to HDFS, the client will be like a RS that writes WAL to HDFS, right? We can use a logic just like distribute log replay to read log entries and use client api to write them to all region servers. Thanks. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635073#comment-15635073
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 4s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 8s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 47 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 53s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
21s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 16s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 43s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m
8s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 9s
{color} | {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 29s
{color} |
[jira] [Updated] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16890: -- Attachment: HBASE-16890-rc-v3.patch Fix the findbugs warnings. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1 > (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch, > AsyncWAL_disruptor_4.patch, HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch, > HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch, > Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 > PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, async.svg, classic.svg, > contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17005) Improve log message in MobFileCache
[ https://issues.apache.org/jira/browse/HBASE-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635059#comment-15635059 ] Jingcheng Du commented on HBASE-17005: -- Thanks for the patch [~huaxiang]! How about to add verbs to the log information? For example "MobFileCache enabled" to "MobFileCache is enabled", and "MobFileCache disabled" to "MobFileCache is disabled". > Improve log message in MobFileCache > --- > > Key: HBASE-17005 > URL: https://issues.apache.org/jira/browse/HBASE-17005 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Attachments: HBASE-17005.master.001.patch, > HBASE-17005.master.002.patch, HBASE-17005.master.003.patch > > > Currently, the MobFileCache prints "MobFileCache is initialized, and the > cache size is 2". When MobFileCache is enabled, it can print out config > params and it is disabled, it prints explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16982) Better integrate Apache CLI in AbstractHBaseTool
[ https://issues.apache.org/jira/browse/HBASE-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635033#comment-15635033 ] Appy commented on HBASE-16982: -- Thanks for the review [~stack] and [~enis] (replied on RB, waiting for your +1). Submitted same patch again for another QA run. Created HBASE-17019 for updating ref guide. > Better integrate Apache CLI in AbstractHBaseTool > > > Key: HBASE-16982 > URL: https://issues.apache.org/jira/browse/HBASE-16982 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16982.master.001.patch, > HBASE-16982.master.002.patch, HBASE-16982.master.003.patch > > > Problem > 1. Inconsistencies in our user facing tools. > - options: > start with single/double dash > words separated by dash or underscore or just joined together > someplace use '=' to separate value, other use space (ExportSnapshot vs > HashTable) > - Description > Manually formatting options and their descriptions in printUsage() > Inconsistant formatting, sometimes even weird. > Incomplete. Sometimes people forget to add new option to description > 2. Manual parsing of options (those huge if-else loops iterating over args) > Solution > Use Apache CLI > - It has various validations for option names which'll fix first set of > issues. > - using AbstractHBaseTool's print usage function will ensure consistent > formatting (although we loose the power to order the options) > - If we enforce the method of defining options as in patch, it's highly > unlikely to forget adding description. > - CLI parses the options for us. > Using Apache CLI when writing new tools is straight forward, but it's not > easy when porting exiting tools since some option names are not valid as per > CLI's validation. > New method, processOldArgs(), will allow to port these tools in a backward > compatible manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17019) Guidelines for naming CLI options of external tools
Appy created HBASE-17019: Summary: Guidelines for naming CLI options of external tools Key: HBASE-17019 URL: https://issues.apache.org/jira/browse/HBASE-17019 Project: HBase Issue Type: Bug Reporter: Appy Assignee: Appy Write a section in hbase book about: - AbstractHBaseTool and how it uses Apache CLI - Basic option naming conventions for consistent user facing tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16982) Better integrate Apache CLI in AbstractHBaseTool
[ https://issues.apache.org/jira/browse/HBASE-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16982: - Attachment: HBASE-16982.master.003.patch > Better integrate Apache CLI in AbstractHBaseTool > > > Key: HBASE-16982 > URL: https://issues.apache.org/jira/browse/HBASE-16982 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16982.master.001.patch, > HBASE-16982.master.002.patch, HBASE-16982.master.003.patch > > > Problem > 1. Inconsistencies in our user facing tools. > - options: > start with single/double dash > words separated by dash or underscore or just joined together > someplace use '=' to separate value, other use space (ExportSnapshot vs > HashTable) > - Description > Manually formatting options and their descriptions in printUsage() > Inconsistant formatting, sometimes even weird. > Incomplete. Sometimes people forget to add new option to description > 2. Manual parsing of options (those huge if-else loops iterating over args) > Solution > Use Apache CLI > - It has various validations for option names which'll fix first set of > issues. > - using AbstractHBaseTool's print usage function will ensure consistent > formatting (although we loose the power to order the options) > - If we enforce the method of defining options as in patch, it's highly > unlikely to forget adding description. > - CLI parses the options for us. > Using Apache CLI when writing new tools is straight forward, but it's not > easy when porting exiting tools since some option names are not valid as per > CLI's validation. > New method, processOldArgs(), will allow to port these tools in a backward > compatible manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17004) Refactor IntegrationTestManyRegions to use @ClassRule for timing out
[
https://issues.apache.org/jira/browse/HBASE-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635008#comment-15635008
]
Hadoop QA commented on HBASE-17004:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 26s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12837014/HBASE-17004.master.002.patch
|
| JIRA Issue | HBASE-17004 |
| Optional Tests | asflicense javac javadoc unit xml compile findbugs
hadoopcheck hbaseanti checkstyle |
| uname | Linux 883328d63b84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / bb0fc6b |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4316/testReport/ |
| modules | C: hbase-it U: hbase-it |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4316/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Refactor IntegrationTestManyRegions to use @ClassRule for timing out
>
>
> Key: HBASE-17004
> URL: https://issues.apache.org/jira/browse/HBASE-17004
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634964#comment-15634964 ] Joep Rottinghuis commented on HBASE-17018: -- [~sduskis] I've made you a watcher of this jira, since we discussed this at the last HBase meetup in NYC. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated HBASE-17018: - Attachment: YARN-4061 HBase requirements for fault tolerant writer.pdf Attaching PDF with additional background, discussion of failure modes we'd like to cover, some thoughts on implementation, and a list of open questions. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17018) Spooling BufferedMutator
Joep Rottinghuis created HBASE-17018: Summary: Spooling BufferedMutator Key: HBASE-17018 URL: https://issues.apache.org/jira/browse/HBASE-17018 Project: HBase Issue Type: New Feature Reporter: Joep Rottinghuis For Yarn Timeline Service v2 we use HBase as a backing store. A big concern we would like to address is what to do if HBase is (temporarily) down, for example in case of an HBase upgrade. Most of the high volume writes will be mostly on a best-effort basis, but occasionally we do a flush. Mainly during application lifecycle events, clients will call a flush on the timeline service API. In order to handle the volume of writes we use a BufferedMutator. When flush gets called on our API, we in turn call flush on the BufferedMutator. We would like our interface to HBase be able to spool the mutations to a filesystems in case of HBase errors. If we use the Hadoop filesystem interface, this can then be HDFS, gcs, s3, or any other distributed storage. The mutations can then later be re-played, for example through a MapReduce job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17004) Refactor IntegrationTestManyRegions to use @ClassRule for timing out
[ https://issues.apache.org/jira/browse/HBASE-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17004: - Description: IntegrationTestManyRegions verifies that many regions get assigned within given time. To do so, it spawns a new thread and uses CountDownLatch.await() to timeout. Replacing this mechanism with junit @ClassRule to timeout the test. > Refactor IntegrationTestManyRegions to use @ClassRule for timing out > > > Key: HBASE-17004 > URL: https://issues.apache.org/jira/browse/HBASE-17004 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-17004.master.001.patch, > HBASE-17004.master.002.patch > > > IntegrationTestManyRegions verifies that many regions get assigned within > given time. To do so, it spawns a new thread and uses CountDownLatch.await() > to timeout. Replacing this mechanism with junit @ClassRule to timeout the > test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17004) Refactor IntegrationTestManyRegions to use @ClassRule for timing out
[ https://issues.apache.org/jira/browse/HBASE-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634949#comment-15634949 ] Appy commented on HBASE-17004: -- Updated the commit message too. > Refactor IntegrationTestManyRegions to use @ClassRule for timing out > > > Key: HBASE-17004 > URL: https://issues.apache.org/jira/browse/HBASE-17004 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-17004.master.001.patch, > HBASE-17004.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634943#comment-15634943
]
Duo Zhang edited comment on HBASE-16890 at 11/4/16 2:04 AM:
So the key point is the HDFS cluster must be single node? I run the WALPE on a
standard HDFS cluster with replication set to 1. I can see this log in the
output
{noformat}
2016-11-03 20:58:16,126 INFO [WALPerfEval.logRoller] wal.AbstractFSWAL: Slow
sync cost: 104 ms, current pipeline:
[DatanodeInfoWithStorage[10.108.18.25:35402,DS-eb75553e-c64d-42a3-ab43-a772a74c7b51,DISK]]
{noformat}
So I think it worked? And I set block size to 1GB and roll size to 512M to
prevent FSHLog roll a new block, the result is still same, FSHLog is the
slowest...
And my HDFS version is 2.7.3.
was (Author: apache9):
So the key point is the HDFS cluster must be single node? I run the WALPE on a
standard HDFS cluster with replication set to 1. I can see this log in the
output
{noformat}
2016-11-03 20:58:16,126 INFO [WALPerfEval.logRoller] wal.AbstractFSWAL: Slow
sync cost: 104 ms, current pipeline:
[DatanodeInfoWithStorage[10.108.18.25:35402,DS-eb75553e-c64d-42a3-ab43-a772a74c7b51,DISK]]
{noformat}
So I think it worked? And I set block size to 1GB and roll size to 512M to
prevent FSHLog roll a new block, the result is still same, FSHLog is the
slowest...
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1
> (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch,
> AsyncWAL_disruptor_4.patch, HBASE-16890-rc-v2.patch,
> HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch,
> Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07
> PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, async.svg, classic.svg,
> contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17004) Refactor IntegrationTestManyRegions to use @ClassRule for timing out
[ https://issues.apache.org/jira/browse/HBASE-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17004: - Attachment: HBASE-17004.master.002.patch > Refactor IntegrationTestManyRegions to use @ClassRule for timing out > > > Key: HBASE-17004 > URL: https://issues.apache.org/jira/browse/HBASE-17004 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-17004.master.001.patch, > HBASE-17004.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634943#comment-15634943
]
Duo Zhang commented on HBASE-16890:
---
So the key point is the HDFS cluster must be single node? I run the WALPE on a
standard HDFS cluster with replication set to 1. I can see this log in the
output
{noformat}
2016-11-03 20:58:16,126 INFO [WALPerfEval.logRoller] wal.AbstractFSWAL: Slow
sync cost: 104 ms, current pipeline:
[DatanodeInfoWithStorage[10.108.18.25:35402,DS-eb75553e-c64d-42a3-ab43-a772a74c7b51,DISK]]
{noformat}
So I think it worked? And I set block size to 1GB and roll size to 512M to
prevent FSHLog roll a new block, the result is still same, FSHLog is the
slowest...
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1
> (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch,
> AsyncWAL_disruptor_4.patch, HBASE-16890-rc-v2.patch,
> HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch,
> Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07
> PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, async.svg, classic.svg,
> contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17016) Reimplement per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17016:
--
Affects Version/s: 1.4.0
2.0.0
Component/s: metrics
> Reimplement per-region latency histogram metrics
>
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> And [~ghelmling] gives us a good candidate to implement pre-region latency
> histograms [HdrHistogram|https://github.com/HdrHistogram/HdrHistogram].
> Let's consider removing the per-region latency histograms and reimplement
> using HdrHistogram.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17016) Reimplement per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634926#comment-15634926
]
Duo Zhang commented on HBASE-17016:
---
Done.
> Reimplement per-region latency histogram metrics
>
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> And [~ghelmling] gives us a good candidate to implement pre-region latency
> histograms [HdrHistogram|https://github.com/HdrHistogram/HdrHistogram].
> Let's consider removing the per-region latency histograms and reimplement
> using HdrHistogram.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17017) Remove the current per-region latency histogram metrics
Duo Zhang created HBASE-17017: - Summary: Remove the current per-region latency histogram metrics Key: HBASE-17017 URL: https://issues.apache.org/jira/browse/HBASE-17017 Project: HBase Issue Type: Sub-task Components: metrics Affects Versions: 2.0.0, 1.4.0 Reporter: Duo Zhang Fix For: 2.0.0, 1.4.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17006) Add names to threads for better debugability of thread dumps
[ https://issues.apache.org/jira/browse/HBASE-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17006: - Resolution: Fixed Status: Resolved (was: Patch Available) > Add names to threads for better debugability of thread dumps > > > Key: HBASE-17006 > URL: https://issues.apache.org/jira/browse/HBASE-17006 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-17006.master.001.patch, > HBASE-17006.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17016) Reimplement per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17016:
--
Description:
Follow up from HBASE-10656, where [~enis] says:
{quote}
the main problem is that we have A LOT of per-region metrics that are latency
histograms. These latency histograms create many many Counter / LongAdder
objects. We should get rid of per-region latencies and maybe look at reducing
the per-region metric overhead.
{quote}
And [~ghelmling] gives us a good candidate to implement pre-region latency
histograms [HdrHistogram|https://github.com/HdrHistogram/HdrHistogram].
Let's consider removing the per-region latency histograms and reimplement using
HdrHistogram.
was:
Follow up from HBASE-10656, where [~enis] says:
{quote}
the main problem is that we have A LOT of per-region metrics that are latency
histograms. These latency histograms create many many Counter / LongAdder
objects. We should get rid of per-region latencies and maybe look at reducing
the per-region metric overhead.
{quote}
And [~ghelmling] gives us a good candidate to implement pre-region latency his
Let's consider removing the per-region latency histograms until we have a low
impact (in time and space) accounting.
> Reimplement per-region latency histogram metrics
>
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> And [~ghelmling] gives us a good candidate to implement pre-region latency
> histograms [HdrHistogram|https://github.com/HdrHistogram/HdrHistogram].
> Let's consider removing the per-region latency histograms and reimplement
> using HdrHistogram.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17006) Add names to threads for better debugability of thread dumps
[ https://issues.apache.org/jira/browse/HBASE-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17006: - Fix Version/s: 1.0.4 > Add names to threads for better debugability of thread dumps > > > Key: HBASE-17006 > URL: https://issues.apache.org/jira/browse/HBASE-17006 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-17006.master.001.patch, > HBASE-17006.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17016) Reimplement per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17016:
--
Description:
Follow up from HBASE-10656, where [~enis] says:
{quote}
the main problem is that we have A LOT of per-region metrics that are latency
histograms. These latency histograms create many many Counter / LongAdder
objects. We should get rid of per-region latencies and maybe look at reducing
the per-region metric overhead.
{quote}
And [~ghelmling] gives us a good candidate to implement pre-region latency his
Let's consider removing the per-region latency histograms until we have a low
impact (in time and space) accounting.
was:
Follow up from HBASE-10656, where [~enis] says:
{quote}
the main problem is that we have A LOT of per-region metrics that are latency
histograms. These latency histograms create many many Counter / LongAdder
objects. We should get rid of per-region latencies and maybe look at reducing
the per-region metric overhead.
{quote}
Let's consider removing the per-region latency histograms until we have a low
impact (in time and space) accounting.
> Reimplement per-region latency histogram metrics
>
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> And [~ghelmling] gives us a good candidate to implement pre-region latency his
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17016) Reimplement per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17016:
--
Summary: Reimplement per-region latency histogram metrics (was: Remove
per-region latency histogram metrics)
> Reimplement per-region latency histogram metrics
>
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17006) Add names to threads for better debugability of thread dumps
[ https://issues.apache.org/jira/browse/HBASE-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17006: - Fix Version/s: 1.1.8 1.2.4 1.4.0 1.3.0 2.0.0 > Add names to threads for better debugability of thread dumps > > > Key: HBASE-17006 > URL: https://issues.apache.org/jira/browse/HBASE-17006 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-17006.master.001.patch, > HBASE-17006.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17016) Remove per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634867#comment-15634867
]
Andrew Purtell commented on HBASE-17016:
Sounds good, please go ahead and do that.
> Remove per-region latency histogram metrics
> ---
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17016) Remove per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634858#comment-15634858
]
Duo Zhang commented on HBASE-17016:
---
So let's change the title to "Reimplement per-region latency histogram
metrics", and the first sub task is remove the current implementation, and also
create other sub tasks to reimplement it with HdrHistogram?
> Remove per-region latency histogram metrics
> ---
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: 14123-master.v35.txt v35. small addendum to HBASE-16978. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16978) Disable backup by default
[ https://issues.apache.org/jira/browse/HBASE-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-16978. Resolution: Fixed > Disable backup by default > - > > Key: HBASE-16978 > URL: https://issues.apache.org/jira/browse/HBASE-16978 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16978-v1.patch, HBASE-16978-v2.patch, > HBASE-16978.addendum, HBASE-16978.addendum.2, HBASE-16978.addendum.3 > > > Currently, we create backup system table on a Master start up (if does not > exists). In deployments where backup is not used this does not make sense. We > should create system table only if backup is enabled and disable it by > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15513) hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default
[ https://issues.apache.org/jira/browse/HBASE-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634744#comment-15634744 ] Vladimir Rodionov commented on HBASE-15513: --- [~saint@gmail.com], [~ramkrishna.s.vasude...@gmail.com]? > hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default > -- > > Key: HBASE-15513 > URL: https://issues.apache.org/jira/browse/HBASE-15513 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15513-v1.patch > > > That results in excessive MemStoreLAB chunk allocations because we can not > reuse them. Not sure, why it has been disabled, by default. May be the code > has not been tested well? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16978) Disable backup by default
[ https://issues.apache.org/jira/browse/HBASE-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16978: -- Attachment: HBASE-16978.addendum.3 cc: [~tedyu] > Disable backup by default > - > > Key: HBASE-16978 > URL: https://issues.apache.org/jira/browse/HBASE-16978 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16978-v1.patch, HBASE-16978-v2.patch, > HBASE-16978.addendum, HBASE-16978.addendum.2, HBASE-16978.addendum.3 > > > Currently, we create backup system table on a Master start up (if does not > exists). In deployments where backup is not used this does not make sense. We > should create system table only if backup is enabled and disable it by > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16978) Disable backup by default
[ https://issues.apache.org/jira/browse/HBASE-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-16978: --- Reopened for small addendum > Disable backup by default > - > > Key: HBASE-16978 > URL: https://issues.apache.org/jira/browse/HBASE-16978 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16978-v1.patch, HBASE-16978-v2.patch, > HBASE-16978.addendum, HBASE-16978.addendum.2 > > > Currently, we create backup system table on a Master start up (if does not > exists). In deployments where backup is not used this does not make sense. We > should create system table only if backup is enabled and disable it by > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16570) Compute region locality in parallel at startup
[ https://issues.apache.org/jira/browse/HBASE-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634710#comment-15634710 ] Gary Helmling commented on HBASE-16570: --- +1 on addendum v4. Thanks for the update, [~aoxiang]. > Compute region locality in parallel at startup > -- > > Key: HBASE-16570 > URL: https://issues.apache.org/jira/browse/HBASE-16570 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16570-master_V1.patch, > HBASE-16570-master_V2.patch, HBASE-16570-master_V3.patch, > HBASE-16570-master_V4.patch, HBASE-16570.branch-1.3-addendum.patch, > HBASE-16570_addnum.patch, HBASE-16570_addnum_v2.patch, > HBASE-16570_addnum_v3.patch, HBASE-16570_addnum_v4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17014) Add clearly marked starting and shutdown log messages for all services.
[ https://issues.apache.org/jira/browse/HBASE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634689#comment-15634689 ] Hudson commented on HBASE-17014: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1910 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1910/]) HBASE-17014 Added clearly marked log messages for start and shutdown of (stack: rev 281bf9a222a3eae006198fb067f227721f723996) * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Add clearly marked starting and shutdown log messages for all services. > --- > > Key: HBASE-17014 > URL: https://issues.apache.org/jira/browse/HBASE-17014 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17014.v1.patch > > > From observing the log messages, clearly marked starting and shutdown > messages for services HMaster, HRegionServer, ThriftServer and RESTServer > will improve log readability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634688#comment-15634688 ] Hudson commented on HBASE-14329: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1910 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1910/]) HBASE-14329 Report region in transition should check all regions in (garyh: rev f61ae6f06dae2c22d0fa3480ec07b1104cb443a1) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15893) Get object
[ https://issues.apache.org/jira/browse/HBASE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634662#comment-15634662 ] Sudeep Sunthankar commented on HBASE-15893: --- Thanks [~enis]. Sorry for mixing up the patches. I will make sure to avoid them going ahead. > Get object > -- > > Key: HBASE-15893 > URL: https://issues.apache.org/jira/browse/HBASE-15893 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Fix For: HBASE-14850 > > Attachments: HBASE-15893.HBASE-14850.v1.patch, > HBASE-15893.HBASE-14850.v2.patch, HBASE-15893.HBASE-14850.v3.patch, > HBASE-15893.HBASE-14850.v4.patch, HBASE-15893.HBASE-14850.v5.patch, > HBASE-15893.HBASE-14850.v6.patch, HBASE-15893.HBASE-14850.v7.patch, > HBASE-15893_v8.patch > > > Patch for creating Get objects. Get objects can be passed to the Table > implementation to fetch results for a given row. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17005) Improve log message in MobFileCache
[
https://issues.apache.org/jira/browse/HBASE-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634581#comment-15634581
]
Hadoop QA commented on HBASE-17005:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestAsyncGetMultiThread |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836967/HBASE-17005.master.003.patch
|
| JIRA Issue | HBASE-17005 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux bc71b6522167 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f61ae6f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4313/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4313/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16964) Successfully archived files are not cleared from compacted store file list if archiving of any file fails
[ https://issues.apache.org/jira/browse/HBASE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634548#comment-15634548 ] Hudson commented on HBASE-16964: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #66 (See [https://builds.apache.org/job/HBase-1.3-JDK8/66/]) HBASE-16964 Always clear successfully archived files from compacted (garyh: rev 1e4d878325754a6b489843a8bb07c928cd4a2a01) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/FailedArchiveException.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Successfully archived files are not cleared from compacted store file list if > archiving of any file fails > - > > Key: HBASE-16964 > URL: https://issues.apache.org/jira/browse/HBASE-16964 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16964.002.patch, HBASE-16964.003.patch, > HBASE-16964.patch > > > In HStore.removeCompactedFiles(), we only clear archived files from > StoreFileManager's list of compactedfiles if _all_ files were archived > successfully. If we encounter an error archiving any of the files, then any > files which were already archived will remain in the list of compactedfiles. > Even worse, this means that all subsequent attempts to archive the list of > compacted files will fail (as the previously successfully archived files > still in the list will now throw FileNotFoundException), and the list of > compactedfiles will never be cleared from that point on. > Finally, when the region closes, we will again throw an exception out of > HStore.removeCompactedFiles(), in this case causing a regionserver abort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17007) Move ZooKeeper logging to its own log file
[ https://issues.apache.org/jira/browse/HBASE-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634534#comment-15634534 ] Enis Soztutar commented on HBASE-17007: --- I have referred to the ZK connection logs before for debugging JAAS setup issues in customer settings and sometimes referred to zk sessionIds etc. I would rather have extra lines than having to manually correlate lines from two different files. It is fine to remove logs which duplicate data like classpath. > Move ZooKeeper logging to its own log file > -- > > Key: HBASE-17007 > URL: https://issues.apache.org/jira/browse/HBASE-17007 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Attachments: > 0001-HBASE-17007-Move-ZooKeeper-logging-to-its-own-log-fi.patch > > > ZooKeeper logging can be too verbose. Lets move ZooKeeper logging to a > different log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17016) Remove per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634512#comment-15634512
]
Enis Soztutar commented on HBASE-17016:
---
bq. So remove now, then maybe put back reimplemented with HdirHistogram, if
workable?
Sounds like a good plan.
> Remove per-region latency histogram metrics
> ---
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths
[
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634502#comment-15634502
]
Hadoop QA commented on HBASE-16941:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 56s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestClientOperationInterrupt |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836957/HBASE-16941.master.003.patch
|
| JIRA Issue | HBASE-16941 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4baa1a15cd0e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f61ae6f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4311/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] [Updated] (HBASE-16960) RegionServer hang when aborting
[ https://issues.apache.org/jira/browse/HBASE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16960: Priority: Critical (was: Major) > RegionServer hang when aborting > --- > > Key: HBASE-16960 > URL: https://issues.apache.org/jira/browse/HBASE-16960 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: 16960.ut.missing.final.piece.txt, > HBASE-16960.branch-1.v1.patch, HBASE-16960.patch, > HBASE-16960_master_v2.patch, HBASE-16960_master_v3.patch, > HBASE-16960_master_v4.patch, RingBufferEventHandler.png, > RingBufferEventHandler_exception.png, SyncFuture.png, > SyncFuture_exception.png, rs1081.jstack > > > We see regionserver hang when aborting several times and cause all regions on > this regionserver out of service and then all affected applications stop > works. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16995) Build client Java API and client protobuf messages
[
https://issues.apache.org/jira/browse/HBASE-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634450#comment-15634450
]
Hadoop QA commented on HBASE-16995:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 38s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 6s
{color} | {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 6s {color} |
{color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 13s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.1 Server=1.12.1 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836973/HBASE-16995.002.patch
|
| JIRA Issue | HBASE-16995 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 991665d13cde 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HBASE-16960) RegionServer hang when aborting
[ https://issues.apache.org/jira/browse/HBASE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16960: Affects Version/s: 1.4.0 1.3.0 2.0.0 > RegionServer hang when aborting > --- > > Key: HBASE-16960 > URL: https://issues.apache.org/jira/browse/HBASE-16960 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: 16960.ut.missing.final.piece.txt, > HBASE-16960.branch-1.v1.patch, HBASE-16960.patch, > HBASE-16960_master_v2.patch, HBASE-16960_master_v3.patch, > HBASE-16960_master_v4.patch, RingBufferEventHandler.png, > RingBufferEventHandler_exception.png, SyncFuture.png, > SyncFuture_exception.png, rs1081.jstack > > > We see regionserver hang when aborting several times and cause all regions on > this regionserver out of service and then all affected applications stop > works. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16960) RegionServer hang when aborting
[ https://issues.apache.org/jira/browse/HBASE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16960: Fix Version/s: 1.4.0 1.3.0 2.0.0 > RegionServer hang when aborting > --- > > Key: HBASE-16960 > URL: https://issues.apache.org/jira/browse/HBASE-16960 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: 16960.ut.missing.final.piece.txt, > HBASE-16960.branch-1.v1.patch, HBASE-16960.patch, > HBASE-16960_master_v2.patch, HBASE-16960_master_v3.patch, > HBASE-16960_master_v4.patch, RingBufferEventHandler.png, > RingBufferEventHandler_exception.png, SyncFuture.png, > SyncFuture_exception.png, rs1081.jstack > > > We see regionserver hang when aborting several times and cause all regions on > this regionserver out of service and then all affected applications stop > works. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16960) RegionServer hang when aborting
[ https://issues.apache.org/jira/browse/HBASE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634437#comment-15634437 ] Mikhail Antonov commented on HBASE-16960: - [~carp84] agreed. thanks for the efforts and patience here! +1 for 1.3.0. > RegionServer hang when aborting > --- > > Key: HBASE-16960 > URL: https://issues.apache.org/jira/browse/HBASE-16960 > Project: HBase > Issue Type: Bug >Reporter: binlijin >Assignee: binlijin > Attachments: 16960.ut.missing.final.piece.txt, > HBASE-16960.branch-1.v1.patch, HBASE-16960.patch, > HBASE-16960_master_v2.patch, HBASE-16960_master_v3.patch, > HBASE-16960_master_v4.patch, RingBufferEventHandler.png, > RingBufferEventHandler_exception.png, SyncFuture.png, > SyncFuture_exception.png, rs1081.jstack > > > We see regionserver hang when aborting several times and cause all regions on > this regionserver out of service and then all affected applications stop > works. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17016) Remove per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634425#comment-15634425
]
Andrew Purtell edited comment on HBASE-17016 at 11/3/16 9:59 PM:
-
bq. We could also look at moving to HdrHistogram for all of our histogram usage.
+1
Really fine grained latency histograms would be nice to have, if cheap enough.
Otherwise, external monitoring based on the Canary can still achieve this, at
cost/benefit of rolling whatever you need.
So remove now, then _maybe_ put back reimplemented with HdirHistogram, if
workable?
was (Author: apurtell):
bq. We could also look at moving to HdrHistogram for all of our histogram usage.
+1
Really fine grained latency histograms would be nice to have, if cheap enough.
Otherwise, external monitoring based on the Canary can still achieve this, at
cost/benefit of rolling whatever you need.
So remove now, then put back reimplemented with HdirHistogram?
> Remove per-region latency histogram metrics
> ---
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17016) Remove per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634425#comment-15634425
]
Andrew Purtell commented on HBASE-17016:
bq. We could also look at moving to HdrHistogram for all of our histogram usage.
+1
Really fine grained latency histograms would be nice to have, if cheap enough.
Otherwise, external monitoring based on the Canary can still achieve this, at
cost/benefit of rolling whatever you need.
So remove now, then put back reimplemented with HdirHistogram?
> Remove per-region latency histogram metrics
> ---
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16964) Successfully archived files are not cleared from compacted store file list if archiving of any file fails
[ https://issues.apache.org/jira/browse/HBASE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634395#comment-15634395 ] Hudson commented on HBASE-16964: FAILURE: Integrated in Jenkins build HBase-1.4 #513 (See [https://builds.apache.org/job/HBase-1.4/513/]) HBASE-16964 Always clear successfully archived files from compacted (garyh: rev 4fe51ebaa794131d7e3e2a042ba85f9d525726c2) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/FailedArchiveException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java > Successfully archived files are not cleared from compacted store file list if > archiving of any file fails > - > > Key: HBASE-16964 > URL: https://issues.apache.org/jira/browse/HBASE-16964 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16964.002.patch, HBASE-16964.003.patch, > HBASE-16964.patch > > > In HStore.removeCompactedFiles(), we only clear archived files from > StoreFileManager's list of compactedfiles if _all_ files were archived > successfully. If we encounter an error archiving any of the files, then any > files which were already archived will remain in the list of compactedfiles. > Even worse, this means that all subsequent attempts to archive the list of > compacted files will fail (as the previously successfully archived files > still in the list will now throw FileNotFoundException), and the list of > compactedfiles will never be cleared from that point on. > Finally, when the region closes, we will again throw an exception out of > HStore.removeCompactedFiles(), in this case causing a regionserver abort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634394#comment-15634394 ] Hudson commented on HBASE-14329: FAILURE: Integrated in Jenkins build HBase-1.4 #513 (See [https://builds.apache.org/job/HBase-1.4/513/]) HBASE-14329 Report region in transition should check all regions in (garyh: rev b4676d51ad8087a7de3f9ea763cf611380aa0431) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16921) TEST: update MOB unit tests to user MasterStorage/ RegionStorage APIs
[
https://issues.apache.org/jira/browse/HBASE-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-16921:
---
Assignee: Sean Busbey
> TEST: update MOB unit tests to user MasterStorage/ RegionStorage APIs
> -
>
> Key: HBASE-16921
> URL: https://issues.apache.org/jira/browse/HBASE-16921
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Umesh Agashe
>Assignee: Sean Busbey
>
> List of files:
> {code}
> hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobFileLink.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/mob/compactions/TestPartitionedMobCompactor.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDeleteMobTable.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobStoreCompaction.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobStoreScanner.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16901) remove directory layout / fs references from bulkload code
[ https://issues.apache.org/jira/browse/HBASE-16901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-16901: --- Assignee: Sean Busbey > remove directory layout / fs references from bulkload code > -- > > Key: HBASE-16901 > URL: https://issues.apache.org/jira/browse/HBASE-16901 > Project: HBase > Issue Type: Sub-task >Reporter: Sean Busbey >Assignee: Sean Busbey > > ensure bulk load implementation relies on MasterStorage / RegionStorage apis > and make changes needed to support that. > hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java > hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java > also probably changes in HRegion -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16898) remove directory layout / fs references from MOB
[
https://issues.apache.org/jira/browse/HBASE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-16898:
---
Assignee: Sean Busbey
> remove directory layout / fs references from MOB
>
>
> Key: HBASE-16898
> URL: https://issues.apache.org/jira/browse/HBASE-16898
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>
> update mob related classes and utilities to use MasterStorage / RegionStorage
> and update those apis as needed to do so.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17016) Remove per-region latency histogram metrics
[
https://issues.apache.org/jira/browse/HBASE-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634364#comment-15634364
]
Gary Helmling commented on HBASE-17016:
---
We could also look at moving to HdrHistogram for all of our histogram usage.
That would eliminate the tons of Counter instances we generate and outperformed
our FastLongHistogram in my testing.
> Remove per-region latency histogram metrics
> ---
>
> Key: HBASE-17016
> URL: https://issues.apache.org/jira/browse/HBASE-17016
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
>
> Follow up from HBASE-10656, where [~enis] says:
> {quote}
> the main problem is that we have A LOT of per-region metrics that are latency
> histograms. These latency histograms create many many Counter / LongAdder
> objects. We should get rid of per-region latencies and maybe look at reducing
> the per-region metric overhead.
> {quote}
> Let's consider removing the per-region latency histograms until we have a low
> impact (in time and space) accounting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634357#comment-15634357 ] Gary Helmling commented on HBASE-10656: --- Yes, it does call reset() after each iteration, because that simulates the actual usage in our metrics code. Benchmarking against a long lived histogram does not represent what actually happens on a live regionserver. If the argument is that we should not be using Counter in histograms, that something we can look at. Again, benchmarks on HBASE-16146 show HdrHistogram implementations outperforming all others. The benchmark there also shows that Counter remains much better than AtomicLong. Where do you see it now performing worse? > high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug > > > Key: HBASE-10656 > URL: https://issues.apache.org/jira/browse/HBASE-10656 > Project: HBase > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt, > HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch, > MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java, > MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java, > output.pdf, output.txt, output2.pdf, output2.txt > > > Cliff's high-scale-lib's Counter is used in important classes (for example, > HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation > detail of the Java standard library and belongs to Oracle (Sun). That > consequently makes HBase depend on the specific JRE Implementation. > To make matters worse, Counter has a bug and you may get wrong result if you > mix a reading method into your logic calling writing methods. > In more detail, I think the bug is caused by reading an internal array field > without resolving memory caching, which is intentional the comment says, but > storing the read result into a volatile field. That field may be not changed > after you can see the true values of the array field, and also may be not > changed after updating the "next" CAT instance's values in some race > condition when extending CAT instance chain. > Anyway, it is possible that you create a new alternative class which only > depends on the standard library. I know Java8 provides its alternative, but > HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16957) Remove directory layout/ filesystem references from Cleaners and a few other modules in master
[
https://issues.apache.org/jira/browse/HBASE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16957:
Fix Version/s: hbase-14439
> Remove directory layout/ filesystem references from Cleaners and a few other
> modules in master
> --
>
> Key: HBASE-16957
> URL: https://issues.apache.org/jira/browse/HBASE-16957
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration, master
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: hbase-14439
>
> Attachments: HBASE-16957-hbase-14439.v1.patch,
> HBASE-16957-hbase-14439.v3.patch, HBASE-16957-jbase-14439.v2.patch
>
>
> Remove directory layout/ filesystem references from Cleaners and a few other
> modules in master
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16957) Remove directory layout/ filesystem references from Cleaners and a few other modules in master
[
https://issues.apache.org/jira/browse/HBASE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16957:
Resolution: Fixed
Release Note:
Incompatible config changes:
* hbase.master.logcleaner.plugins default value that users should always add
when customizing changes from
org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner to
org.apache.hadoop.hbase.fs.legacy.cleaner.TimeToLiveLogCleaner
* hbase.master.hfilecleaner.plugins default value that users should always add
when customizing changes from
org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner to
org.apache.hadoop.hbase.fs.legacy.cleaner.TimeToLiveHFileCleaner
Downstream users who have set custom plugins for either log cleaning or hfile
cleaning should be sure to update their configs to use the new names of the
respective time to live implementations.
Status: Resolved (was: Patch Available)
> Remove directory layout/ filesystem references from Cleaners and a few other
> modules in master
> --
>
> Key: HBASE-16957
> URL: https://issues.apache.org/jira/browse/HBASE-16957
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration, master
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Attachments: HBASE-16957-hbase-14439.v1.patch,
> HBASE-16957-hbase-14439.v3.patch, HBASE-16957-jbase-14439.v2.patch
>
>
> Remove directory layout/ filesystem references from Cleaners and a few other
> modules in master
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16995) Build client Java API and client protobuf messages
[ https://issues.apache.org/jira/browse/HBASE-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634298#comment-15634298 ] Ted Yu commented on HBASE-16995: Please put on review board when the patch is ready for review. > Build client Java API and client protobuf messages > -- > > Key: HBASE-16995 > URL: https://issues.apache.org/jira/browse/HBASE-16995 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: HBASE-16995.001.patch, HBASE-16995.002.patch > > > Extend the existing Java API and protobuf messages to allow the client to set > filesystem-use quotas via the Master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17007) Move ZooKeeper logging to its own log file
[ https://issues.apache.org/jira/browse/HBASE-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634294#comment-15634294 ] Esteban Gutierrez commented on HBASE-17007: --- [~enis] I think what [~stack] said is pretty much one of the main problems with the current logging that we have in the RSs. I think it doesn't hurt doing this change and helps to reduce the clutter specially during startup of the RS. > Move ZooKeeper logging to its own log file > -- > > Key: HBASE-17007 > URL: https://issues.apache.org/jira/browse/HBASE-17007 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Attachments: > 0001-HBASE-17007-Move-ZooKeeper-logging-to-its-own-log-fi.patch > > > ZooKeeper logging can be too verbose. Lets move ZooKeeper logging to a > different log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16995) Build client Java API and client protobuf messages
[ https://issues.apache.org/jira/browse/HBASE-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-16995: --- Attachment: HBASE-16995.002.patch .002 More unit tests and adds some missing deserialization logic that I noticed was missing when working on HBASE-16996. > Build client Java API and client protobuf messages > -- > > Key: HBASE-16995 > URL: https://issues.apache.org/jira/browse/HBASE-16995 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: HBASE-16995.001.patch, HBASE-16995.002.patch > > > Extend the existing Java API and protobuf messages to allow the client to set > filesystem-use quotas via the Master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16996) Implement storage/retrieval of filesystem-use quotas into quota table
[
https://issues.apache.org/jira/browse/HBASE-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-16996:
---
Attachment: HBASE-16996.001.patch
.001 Some basic CRUD operations on hbase:quota via the public API and unit
tests.
> Implement storage/retrieval of filesystem-use quotas into quota table
> -
>
> Key: HBASE-16996
> URL: https://issues.apache.org/jira/browse/HBASE-16996
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0
>
> Attachments: HBASE-16996.001.patch
>
>
> Provide read/write API for accessing the new filesystem-usage quotas in the
> existing {{hbase:quota}} table.
> Make sure that both the client can read quotas the quotas in the table as
> well as the Master can perform the necessary update/delete actions per the
> quota RPCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16996) Implement storage/retrieval of filesystem-use quotas into quota table
[
https://issues.apache.org/jira/browse/HBASE-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-16996:
---
Fix Version/s: 2.0.0
> Implement storage/retrieval of filesystem-use quotas into quota table
> -
>
> Key: HBASE-16996
> URL: https://issues.apache.org/jira/browse/HBASE-16996
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0
>
>
> Provide read/write API for accessing the new filesystem-usage quotas in the
> existing {{hbase:quota}} table.
> Make sure that both the client can read quotas the quotas in the table as
> well as the Master can perform the necessary update/delete actions per the
> quota RPCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12894) Upgrade Jetty to 9.2.6
[
https://issues.apache.org/jira/browse/HBASE-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634270#comment-15634270
]
Sean Busbey commented on HBASE-12894:
-
Please make sure you've checked all the new dependency licenses. For example:
{code}
+
+
+ org.eclipse.jetty.toolchain
+ jetty-schemas
+
+
+
+ Apache License, Version 2.0
+ http://www.apache.org/licenses/LICENSE-2.0.txt
+ repo
+
+
+
+
{code}
The [jetty project
NOTICE|https://github.com/eclipse/jetty.project/blob/jetty-9.3.x/NOTICE.txt]
calls out this dependency:
{code}
--
Mortbay
The following artifacts are CDDL + GPLv2 with classpath exception.
https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html
org.eclipse.jetty.toolchain:jetty-schemas
{code}
> Upgrade Jetty to 9.2.6
> --
>
> Key: HBASE-12894
> URL: https://issues.apache.org/jira/browse/HBASE-12894
> Project: HBase
> Issue Type: Improvement
> Components: REST, UI
>Affects Versions: 0.98.0
>Reporter: Rick Hallihan
>Assignee: Guang Yang
>Priority: Critical
> Labels: MicrosoftSupport
> Fix For: 2.0.0
>
> Attachments: HBASE-12894_Jetty9_v0.patch,
> HBASE-12894_Jetty9_v1.patch, HBASE-12894_Jetty9_v1.patch,
> HBASE-12894_Jetty9_v2.patch, HBASE-12894_Jetty9_v3.patch,
> HBASE-12894_Jetty9_v4.patch, HBASE-12894_Jetty9_v5.patch,
> HBASE-12894_Jetty9_v6.patch, HBASE-12894_Jetty9_v7.patch,
> dependency_list_after, dependency_list_before
>
>
> The Jetty component that is used for the HBase Stargate REST endpoint is
> version 6.1.26 and is fairly outdated. We recently had a customer inquire
> about enabling cross-origin resource sharing (CORS) for the REST endpoint and
> found that this older version does not include the necessary filter or
> configuration options, highlighted at:
> http://wiki.eclipse.org/Jetty/Feature/Cross_Origin_Filter
> The Jetty project has had significant updates through versions 7, 8 and 9,
> including a transition to be an Eclipse subproject, so updating to the latest
> version may be non-trivial. The last update to the Jetty component in
> https://issues.apache.org/jira/browse/HBASE-3377 was a minor version update
> and did not require significant work. This update will include a package
> namespace update so there will likely be a larger number of required changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17007) Move ZooKeeper logging to its own log file
[ https://issues.apache.org/jira/browse/HBASE-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634259#comment-15634259 ] Esteban Gutierrez commented on HBASE-17007: --- [~enis] I think what [~stack] said is pretty much one of the main problems with the current logging that we have in the RSs. I think it doesn't hurt doing this change and helps to reduce the clutter specially during startup of the RS. > Move ZooKeeper logging to its own log file > -- > > Key: HBASE-17007 > URL: https://issues.apache.org/jira/browse/HBASE-17007 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Attachments: > 0001-HBASE-17007-Move-ZooKeeper-logging-to-its-own-log-fi.patch > > > ZooKeeper logging can be too verbose. Lets move ZooKeeper logging to a > different log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17005) Improve log message in MobFileCache
[ https://issues.apache.org/jira/browse/HBASE-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17005: - Attachment: HBASE-17005.master.003.patch > Improve log message in MobFileCache > --- > > Key: HBASE-17005 > URL: https://issues.apache.org/jira/browse/HBASE-17005 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Attachments: HBASE-17005.master.001.patch, > HBASE-17005.master.002.patch, HBASE-17005.master.003.patch > > > Currently, the MobFileCache prints "MobFileCache is initialized, and the > cache size is 2". When MobFileCache is enabled, it can print out config > params and it is disabled, it prints explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17005) Improve log message in MobFileCache
[ https://issues.apache.org/jira/browse/HBASE-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634176#comment-15634176 ] huaxiang sun commented on HBASE-17005: -- Thanks [~jmhsieh] and [~jingcheng...@intel.com]. Uploaded a new patch. The output is something like: 2016-11-03 13:34:15,783 INFO [main] mob.MobFileCache(121): MobFileCache enabled with cacheSize=2111, evictPeriods=3600sec, evictRemainRatio=0.5 > Improve log message in MobFileCache > --- > > Key: HBASE-17005 > URL: https://issues.apache.org/jira/browse/HBASE-17005 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Attachments: HBASE-17005.master.001.patch, > HBASE-17005.master.002.patch, HBASE-17005.master.003.patch > > > Currently, the MobFileCache prints "MobFileCache is initialized, and the > cache size is 2". When MobFileCache is enabled, it can print out config > params and it is disabled, it prints explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16964) Successfully archived files are not cleared from compacted store file list if archiving of any file fails
[ https://issues.apache.org/jira/browse/HBASE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634106#comment-15634106 ] Hudson commented on HBASE-16964: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #58 (See [https://builds.apache.org/job/HBase-1.3-JDK7/58/]) HBASE-16964 Always clear successfully archived files from compacted (garyh: rev 1e4d878325754a6b489843a8bb07c928cd4a2a01) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/FailedArchiveException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java > Successfully archived files are not cleared from compacted store file list if > archiving of any file fails > - > > Key: HBASE-16964 > URL: https://issues.apache.org/jira/browse/HBASE-16964 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16964.002.patch, HBASE-16964.003.patch, > HBASE-16964.patch > > > In HStore.removeCompactedFiles(), we only clear archived files from > StoreFileManager's list of compactedfiles if _all_ files were archived > successfully. If we encounter an error archiving any of the files, then any > files which were already archived will remain in the list of compactedfiles. > Even worse, this means that all subsequent attempts to archive the list of > compacted files will fail (as the previously successfully archived files > still in the list will now throw FileNotFoundException), and the list of > compactedfiles will never be cleared from that point on. > Finally, when the region closes, we will again throw an exception out of > HStore.removeCompactedFiles(), in this case causing a regionserver abort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17014) Add clearly marked starting and shutdown log messages for all services.
[ https://issues.apache.org/jira/browse/HBASE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634077#comment-15634077 ] Umesh Agashe commented on HBASE-17014: -- They are similar but not same. Hadoop's are multiline. For Hbase, we can have terse. > Add clearly marked starting and shutdown log messages for all services. > --- > > Key: HBASE-17014 > URL: https://issues.apache.org/jira/browse/HBASE-17014 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17014.v1.patch > > > From observing the log messages, clearly marked starting and shutdown > messages for services HMaster, HRegionServer, ThriftServer and RESTServer > will improve log readability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16960) RegionServer hang when aborting
[
https://issues.apache.org/jira/browse/HBASE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634074#comment-15634074
]
Yu Li commented on HBASE-16960:
---
Thanks for checking this [~mantonov]
bq. How bad is this for you? How frequently you see that Yu Li and binlijin?
It's really bad, as mentioned in the JIRA description:
{noformat}
We see regionserver hang when aborting several times and cause all regions on
this regionserver out of service and then all affected applications stop works.
{noformat}
This happens not that frequently but still some, like in our case we saw such
hang before addressing it twice a week or so.
bq. Any follow-ups on that?...
Not yet, but will do. This is another topic and will be big I assume...
bq. Thinking to move it to 1.3.1, where I'd bring those changed and bake in.
Thoughts (that depends on how bad this issue is) ?
I'd suggest to have it in 1.3.0 since it will cause severe problem and with
some frequency. The issue is already reproduced and covered in the UT case here
so it's relatively safe. What's more, we already have this fix running online
here in Alibaba with really high workload for about one week and no regression
issue observed, and I guess this makes it safer to bring in (Smile)? Thanks.
> RegionServer hang when aborting
> ---
>
> Key: HBASE-16960
> URL: https://issues.apache.org/jira/browse/HBASE-16960
> Project: HBase
> Issue Type: Bug
>Reporter: binlijin
>Assignee: binlijin
> Attachments: 16960.ut.missing.final.piece.txt,
> HBASE-16960.branch-1.v1.patch, HBASE-16960.patch,
> HBASE-16960_master_v2.patch, HBASE-16960_master_v3.patch,
> HBASE-16960_master_v4.patch, RingBufferEventHandler.png,
> RingBufferEventHandler_exception.png, SyncFuture.png,
> SyncFuture_exception.png, rs1081.jstack
>
>
> We see regionserver hang when aborting several times and cause all regions on
> this regionserver out of service and then all affected applications stop
> works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16960) RegionServer hang when aborting
[
https://issues.apache.org/jira/browse/HBASE-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634074#comment-15634074
]
Yu Li edited comment on HBASE-16960 at 11/3/16 8:04 PM:
Thanks for checking this [~mantonov]
bq. How bad is this for you? How frequently you see that Yu Li and binlijin?
It's really bad, as mentioned in the JIRA description:
{noformat}
We see regionserver hang when aborting several times and cause all regions on
this
regionserver out of service and then all affected applications stop works.
{noformat}
This happens not that frequently but still some, like in our case we saw such
hang before addressing it twice a week or so.
bq. Any follow-ups on that?...
Not yet, but will do. This is another topic and will be big I assume...
bq. Thinking to move it to 1.3.1, where I'd bring those changed and bake in.
Thoughts (that depends on how bad this issue is) ?
I'd suggest to have it in 1.3.0 since it will cause severe problem and with
some frequency. The issue is already reproduced and covered in the UT case here
so it's relatively safe. What's more, we already have this fix running online
here in Alibaba with really high workload for about one week and no regression
issue observed, and I guess this makes it safer to bring in (Smile)? Thanks.
was (Author: carp84):
Thanks for checking this [~mantonov]
bq. How bad is this for you? How frequently you see that Yu Li and binlijin?
It's really bad, as mentioned in the JIRA description:
{noformat}
We see regionserver hang when aborting several times and cause all regions on
this regionserver out of service and then all affected applications stop works.
{noformat}
This happens not that frequently but still some, like in our case we saw such
hang before addressing it twice a week or so.
bq. Any follow-ups on that?...
Not yet, but will do. This is another topic and will be big I assume...
bq. Thinking to move it to 1.3.1, where I'd bring those changed and bake in.
Thoughts (that depends on how bad this issue is) ?
I'd suggest to have it in 1.3.0 since it will cause severe problem and with
some frequency. The issue is already reproduced and covered in the UT case here
so it's relatively safe. What's more, we already have this fix running online
here in Alibaba with really high workload for about one week and no regression
issue observed, and I guess this makes it safer to bring in (Smile)? Thanks.
> RegionServer hang when aborting
> ---
>
> Key: HBASE-16960
> URL: https://issues.apache.org/jira/browse/HBASE-16960
> Project: HBase
> Issue Type: Bug
>Reporter: binlijin
>Assignee: binlijin
> Attachments: 16960.ut.missing.final.piece.txt,
> HBASE-16960.branch-1.v1.patch, HBASE-16960.patch,
> HBASE-16960_master_v2.patch, HBASE-16960_master_v3.patch,
> HBASE-16960_master_v4.patch, RingBufferEventHandler.png,
> RingBufferEventHandler_exception.png, SyncFuture.png,
> SyncFuture_exception.png, rs1081.jstack
>
>
> We see regionserver hang when aborting several times and cause all regions on
> this regionserver out of service and then all affected applications stop
> works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16956) Refactor FavoredNodePlan to use regionNames as keys
[ https://issues.apache.org/jira/browse/HBASE-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16956: - Status: Open (was: Patch Available) > Refactor FavoredNodePlan to use regionNames as keys > --- > > Key: HBASE-16956 > URL: https://issues.apache.org/jira/browse/HBASE-16956 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16956.master.001.patch, > HBASE-16956.master.002.patch, HBASE-16956.master.003.patch, > HBASE-16956.master.004.patch, HBASE-16956.master.005.patch > > > We would like to rely on the FNPlan cache whether a region is offline or not. > Sticking to regionNames as keys makes that possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Attachment: HBASE-16941.master.003.patch > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Status: Patch Available (was: Open) Addressed pre-commit build warnings and failures and updated the reviewboard link. > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17014) Add clearly marked starting and shutdown log messages for all services.
[
https://issues.apache.org/jira/browse/HBASE-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634011#comment-15634011
]
Hadoop QA commented on HBASE-17014:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 14s {color}
| {color:red} hbase-rest in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.rest.TestTableResource |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestSnapshotMetadata |
| | org.apache.hadoop.hbase.client.TestReplicaWithCluster |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
| | org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836903/HBASE-17014.v1.patch |
| JIRA Issue | HBASE-17014 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d2572711764c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
[jira] [Commented] (HBASE-16413) Add apache-hbase.slack.com #users channel to ref guide
[
https://issues.apache.org/jira/browse/HBASE-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633978#comment-15633978
]
Hudson commented on HBASE-16413:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1909 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1909/])
HBASE-16413 Add apache-hbase.slack.com #users channel to ref guide (stack: rev
de97933aa5e64b315a220919566ffde1a8c273f7)
* (edit) src/main/site/xdoc/index.xml
* (edit) src/main/asciidoc/_chapters/troubleshooting.adoc
* (edit) src/main/asciidoc/_chapters/developer.adoc
> Add apache-hbase.slack.com #users channel to ref guide
> --
>
> Key: HBASE-16413
> URL: https://issues.apache.org/jira/browse/HBASE-16413
> Project: HBase
> Issue Type: Task
> Components: documentation
>Reporter: Dima Spivak
>Assignee: stack
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0
>
>
> We have a user channel on Slack at [http://apache-hbase.slack.com], but no
> reference to it in the [Getting Involved
> section|https://hbase.apache.org/book.html#getting.involved] of our ref
> guide. We should fix that and probably note that new users who want access
> and don't have an {{apache.org}} email address should request an invite by
> emailing [dev@|mailto:d...@hbase.apache.org].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Status: Open (was: Patch Available) > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16964) Successfully archived files are not cleared from compacted store file list if archiving of any file fails
[ https://issues.apache.org/jira/browse/HBASE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633977#comment-15633977 ] Hudson commented on HBASE-16964: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1909 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1909/]) HBASE-16964 Always clear successfully archived files from compacted (garyh: rev e820e64272220cccfe403814606c1f0b17f287ca) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionArchiveIOException.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/FailedArchiveException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java > Successfully archived files are not cleared from compacted store file list if > archiving of any file fails > - > > Key: HBASE-16964 > URL: https://issues.apache.org/jira/browse/HBASE-16964 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16964.002.patch, HBASE-16964.003.patch, > HBASE-16964.patch > > > In HStore.removeCompactedFiles(), we only clear archived files from > StoreFileManager's list of compactedfiles if _all_ files were archived > successfully. If we encounter an error archiving any of the files, then any > files which were already archived will remain in the list of compactedfiles. > Even worse, this means that all subsequent attempts to archive the list of > compacted files will fail (as the previously successfully archived files > still in the list will now throw FileNotFoundException), and the list of > compactedfiles will never be cleared from that point on. > Finally, when the region closes, we will again throw an exception out of > HStore.removeCompactedFiles(), in this case causing a regionserver abort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17015) Multipile identical CacheConfig log messages in a log file and refactoring around CacheConfig inheritance
[
https://issues.apache.org/jira/browse/HBASE-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634000#comment-15634000
]
stack commented on HBASE-17015:
---
We CacheConfig whenever we open a Region (twice). We do it (once?) when we
compact. See below. Yeah, mob is using cache config but it looks like it
shouldn't be using it. Thanks [~uagashe]
307816 2016-10-13 12:24:45,306 INFO
[regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020-longCompactions-1476317858127]
hfile.CacheConfig: blockCache=LruBlockCache{blockCount=12,
currentSize=23198408, freeSize=22529416504, maxSize=22552614912,
heapSize=23198408, minSize=21424984064, minFactor=0.95, multiSize=10712492032,
multiFactor=0.5, singleSize=5356246016, singleFactor=0.25},
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cach
eBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
prefetchOnOpen=false
307817 2016-10-13 12:24:45,306 INFO
[regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020-shortCompactions-1476386685271]
hfile.CacheConfig: blockCache=LruBlockCache{blockCount=12,
currentSize=23198408, freeSize=22529416504, maxSize=22552614912,
heapSize=23198408, minSize=21424984064, minFactor=0.95, multiSize=10712492032,
multiFactor=0.5, singleSize=5356246016, singleFactor=0.25},
cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cac
heBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
prefetchOnOpen=false
307818 2016-10-13 12:24:45,323 INFO
[regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020-longCompactions-1476317858127]
throttle.PressureAwareThroughputController:
hbase:meta,,1.1588230740#table#compaction#5 average throughput is
unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations
remaining, total limit is 10.00 MB/sec
> Multipile identical CacheConfig log messages in a log file and refactoring
> around CacheConfig inheritance
> -
>
> Key: HBASE-17015
> URL: https://issues.apache.org/jira/browse/HBASE-17015
> Project: HBase
> Issue Type: Improvement
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>
> There are two issues here:
> 1. CacheConfig is logged multiple times (some times on adjacent lines) in log
> file. The reason for this is: {code} LOG.info(this) {code} in CacheConfig
> constructor. So when CacheConfig and any derived class is instantiated
> message appears multiple times in a log. The message itself doesn't have any
> context and look identical. Logging CacheConfig from outside constructor will
> fix this issue.
> 2. During debugging its noticed that CacheConfig is meant for in memory block
> cache but is used for different caches (in some cases for non in memory block
> caches). This can be improved by refactoring the inheritance of CacheConfig.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17007) Move ZooKeeper logging to its own log file
[ https://issues.apache.org/jira/browse/HBASE-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633961#comment-15633961 ] stack commented on HBASE-17007: --- We are referring to the dup log of environment and classpath all at INFO level. Four lines each time it makes a connection: 487532 2016-11-01 23:06:35,577 INFO [regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020] zookeeper.ZooKeeper: Initiating client connection, connectString=ve0524.halxg.cloudera.com: sessionTimeout=9 watcher=org.apache.hadoo p.hbase.zookeeper.PendingWatcher@76ae1cc9 487533 2016-11-01 23:06:35,578 INFO [regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020-SendThread(ve0524.halxg.cloudera.com:)] zookeeper.ClientCnxn: Opening socket connection to server ve0524.halxg.cloudera.com/10.17.240.20:2 222. Will not attempt to authenticate using SASL (unknown error) 487534 2016-11-01 23:06:35,578 INFO [regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020-SendThread(ve0524.halxg.cloudera.com:)] zookeeper.ClientCnxn: Socket connection established to ve0524.halxg.cloudera.com/10.17.240.20: , initiating session 487535 2016-11-01 23:06:35,587 INFO [regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020-SendThread(ve0524.halxg.cloudera.com:)] zookeeper.ClientCnxn: Session establishment complete on server ve0524.halxg.cloudera.com/10.17.240 .20:, sessionid = 0x15823a5618b0018, negotiated timeout = 9 In particular, a new user starting up a standalone instance gets 200 lines of logs just on startup of which > 10% are zk for the most part duplicating what we've already said. Doesn't look the best. > Move ZooKeeper logging to its own log file > -- > > Key: HBASE-17007 > URL: https://issues.apache.org/jira/browse/HBASE-17007 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Attachments: > 0001-HBASE-17007-Move-ZooKeeper-logging-to-its-own-log-fi.patch > > > ZooKeeper logging can be too verbose. Lets move ZooKeeper logging to a > different log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17009) Revisiting the removement of managed connection and connection caching
[
https://issues.apache.org/jira/browse/HBASE-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633929#comment-15633929
]
Yu Li commented on HBASE-17009:
---
Thank you for the reference to HBASE-16713 [~enis], let me check it carefully
and will be back later today.
> Revisiting the removement of managed connection and connection caching
> --
>
> Key: HBASE-17009
> URL: https://issues.apache.org/jira/browse/HBASE-17009
> Project: HBase
> Issue Type: Task
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
>
> In HBASE-13197 we have done lots of good cleanups for Connection API, but
> among which HBASE-13252 dropped the feature of managed connection and
> connection caching, and this JIRA propose to have a revisit on this decision
> for below reasons.
> Assume we have a long running process with multiple threads accessing HBase
> (a common case for streaming application), let's see what happens previously
> and now.
> Previously:
> User could create an HTable instance whenever they want w/o worrying about
> the underlying connections because HBase client will mange it automatically,
> say no matter how many threads there will be only one Connection instance
> {code}
> @Deprecated
> public HTable(Configuration conf, final TableName tableName)
> throws IOException {
> ...
> this.connection = ConnectionManager.getConnectionInternal(conf);
> ...
> }
> static ClusterConnection getConnectionInternal(final Configuration conf)
> throws IOException {
> HConnectionKey connectionKey = new HConnectionKey(conf);
> synchronized (CONNECTION_INSTANCES) {
> HConnectionImplementation connection =
> CONNECTION_INSTANCES.get(connectionKey);
> if (connection == null) {
> connection = (HConnectionImplementation)createConnection(conf, true);
> CONNECTION_INSTANCES.put(connectionKey, connection);
> } else if (connection.isClosed()) {
> ConnectionManager.deleteConnection(connectionKey, true);
> connection = (HConnectionImplementation)createConnection(conf, true);
> CONNECTION_INSTANCES.put(connectionKey, connection);
> }
> connection.incCount();
> return connection;
> }
> }
> {code}
> Now:
> User has to create the connection by themselves, using below codes like
> indicated in our recommendations
> {code}
> Connection connection = ConnectionFactory.createConnection(conf);
> Table table = connection.getTable(tableName);
> {code}
> And they must make sure *only one* single connection created in one *process*
> instead of creating HTable instance freely, or else there might be many
> connections setup to zookeeper/RS with multiple threads. Also user might ask
> "when I should close the connection I close?" and the answer is "make sure
> don't close it until the *process* shutdown"
> So now we have much more things for user to "Make sure", but custom is
> something hard to change. User used to create table instance in each thread
> (according to which table to access per requested) so probably they will
> still create connections everywhere, and then operators will have to crazily
> resolve all kinds of problems...
> So I'm proposing to add back the managed connection and connection caching
> support. IMHO it's something good and ever existed in our implementation, so
> let's bring it back and save the workload for operators when they decided to
> upgrade from 1.x to 2.x
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17013) Add constructor to RowMutations for initializing the capacity of internal list
[
https://issues.apache.org/jira/browse/HBASE-17013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633891#comment-15633891
]
Hadoop QA commented on HBASE-17013:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 32s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836885/HBASE-17013.v0.patch |
| JIRA Issue | HBASE-17013 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3a3cff7482c1 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master /
[jira] [Commented] (HBASE-17007) Move ZooKeeper logging to its own log file
[
https://issues.apache.org/jira/browse/HBASE-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633788#comment-15633788
]
Enis Soztutar commented on HBASE-17007:
---
Are you guys talking about ZK server logs in case of mini-cluster / single
master or ZK client logs? I would like to keep the ZK Client logs together with
the region server logs. The ZK server logs in case we start the process can go
into a different log.
If you are talking about log lines like this:
{code}
2016-06-16 19:14:56,931 INFO [ProcessThread(sid:0 cport:2181):]
server.PrepRequestProcessor: Got user-level KeeperException when processing
sessionid:0x1555c23be730001 type:create cxid:0x7 zxid:0x14 tx
{code}
these are coming from the server side.
I just checked RS logs, but did not see anything like an exception or an entry
logged frequently from the ZK Client.
> Move ZooKeeper logging to its own log file
> --
>
> Key: HBASE-17007
> URL: https://issues.apache.org/jira/browse/HBASE-17007
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Trivial
> Attachments:
> 0001-HBASE-17007-Move-ZooKeeper-logging-to-its-own-log-fi.patch
>
>
> ZooKeeper logging can be too verbose. Lets move ZooKeeper logging to a
> different log file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-14329: -- Resolution: Fixed Fix Version/s: (was: 1.3.1) 1.4.0 2.0.0 Status: Resolved (was: Patch Available) Committed to branch-1 and master. > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633758#comment-15633758 ] Gary Helmling commented on HBASE-14329: --- +1 on the patch. The current handling is not exactly broken since the region open case (splits are handled as a separate event) only passes one region. But this will safeguard us if it changes in the future. Thanks for the patch, [~elizondo93]. > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
