[jira] [Created] (HBASE-19705) Remove AMv2 dependency on advancing clock
Sergey Soldatov created HBASE-19705: --- Summary: Remove AMv2 dependency on advancing clock Key: HBASE-19705 URL: https://issues.apache.org/jira/browse/HBASE-19705 Project: HBase Issue Type: Bug Components: amv2 Affects Versions: 2.0.0-beta-1 Reporter: Sergey Soldatov As per discussion on dev list it would be nice to remove the dependency of AMv2 on advancing clock. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19506) Support variable sized chunks from ChunkCreator
[ https://issues.apache.org/jira/browse/HBASE-19506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310857#comment-16310857 ] Anoop Sam John commented on HBASE-19506: Very happy to see u coming up with real math.. The under utilization was my concern from day 1.. I like the way u reasoning out things. bq.Where does the number 52800 come from ? A math issue? 12800 * 5 is what she was saying. So the idea is to have a smaller sized chunk's pool and use that for the index meta data. When we use CCM this pool will be ON and will be used. bq.Let's say chunks of 256KB size So when one in memory flush or compaction require more size than this (for the index meta data), we will use more than one chunk. Correct? Only thing is how we decide the #BBs in this new pool. On what basis. Ya the idea seems interesting. Hope u will come out with a more detailed plan. Good one... > Support variable sized chunks from ChunkCreator > --- > > Key: HBASE-19506 > URL: https://issues.apache.org/jira/browse/HBASE-19506 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky > > When CellChunkMap is created it allocates a special index chunk (or chunks) > where array of cell-representations is stored. When the number of > cell-representations is small, it is preferable to allocate a chunk smaller > than a default value which is 2MB. > On the other hand, those "non-standard size" chunks can not be used in pool. > On-demand allocations in off-heap are costly. So this JIRA is about to > investigate the trade of between memory usage and the final performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19634) Add permission check for executeProcedures in AccessController
[ https://issues.apache.org/jira/browse/HBASE-19634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310843#comment-16310843 ] Duo Zhang commented on HBASE-19634: --- Will commit shortly. > Add permission check for executeProcedures in AccessController > -- > > Key: HBASE-19634 > URL: https://issues.apache.org/jira/browse/HBASE-19634 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19634-HBASE-19397-v1.patch, > HBASE-19634-HBASE-19397-v1.patch, HBASE-19634-HBASE-19397-v2.patch, > HBASE-19634-HBASE-19397.patch > > > This is important, the actual refresh on RS is trigger by the > executeProcedure call and it will pass some information. These information > should not be fully trusted since anyone can all this method. We need to make > sure that the actual data/state for a replication peer is always loaded from > the replication storage, not from the parameter of the executeProcedure call. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19588) Additional jar dependencies needed for mapreduce PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310825#comment-16310825
]
Hudson commented on HBASE-19588:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4340 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4340/])
HBASE-19588 Additional jar dependencies needed for mapreduce (stack: rev
4ba741674d623309e0ff6cd37d2b53ab7c6d7398)
* (edit)
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
> Additional jar dependencies needed for mapreduce PerformanceEvaluation
> --
>
> Key: HBASE-19588
> URL: https://issues.apache.org/jira/browse/HBASE-19588
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.4.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Minor
> Fix For: 1.4.1, 2.0.0-beta-2
>
> Attachments: HBASE-19588.branch-1.4.patch
>
>
> I have a unit test that runs a simple PerformanceEvaluation test to make sure
> things are basically working
> {noformat}
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=5
> sequentialWrite 1
> {noformat}
> This test runs against Hadoop 2.7.0 and works against all past versions
> 0.99.0 and up. It broke with 1.4.0 with the following error.
> {noformat}
> 2017-12-21 13:49:40,974 INFO [main] mapreduce.Job: Task Id :
> attempt_1513892752187_0002_m_04_2, Status : FAILED
> Error: java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:297)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:250)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> ... 12 more
> Caused by: java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeper.(MetricsZooKeeper.java:38)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:130)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:143)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:181)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:155)
> at
> org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.(ZooKeeperKeepAliveConnection.java:43)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1737)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:945)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:721)
> ... 17 more
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not
[jira] [Commented] (HBASE-19490) Rare failure in TestRateLimiter
[ https://issues.apache.org/jira/browse/HBASE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310824#comment-16310824 ] Hudson commented on HBASE-19490: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4340 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4340/]) HBASE-19490 Rare failure in TestRateLimiter (chia7712: rev 338a74e73705fd7c80111ade47345b2a6efe11e7) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java > Rare failure in TestRateLimiter > --- > > Key: HBASE-19490 > URL: https://issues.apache.org/jira/browse/HBASE-19490 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Chia-Ping Tsai > Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7, 2.0.0-beta-1 > > Attachments: HBASE-19490.branch-1.v0.patch, HBASE-19490.v0.patch > > > Maybe we aren't waiting long enough for a slow executor? Or it is some kind > of race. Test usually passes. > [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.01 > s <<< FAILURE! - in org.apache.hadoop.hbase.quotas.TestRateLimiter > [ERROR] > testOverconsumptionFixedIntervalRefillStrategy(org.apache.hadoop.hbase.quotas.TestRateLimiter) > Time elapsed: 0.028 s <<< FAILURE! > java.lang.AssertionError: expected:<1000> but was:<999> > at > org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:122) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19473) Miscellaneous changes to ClientScanner
[
https://issues.apache.org/jira/browse/HBASE-19473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310826#comment-16310826
]
Hudson commented on HBASE-19473:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4340 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4340/])
HBASE-19473 Miscellaneous changes to ClientScanner (appy: rev
2bd259b445971131526a5e6580363c92dc597b10)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
> Miscellaneous changes to ClientScanner
> --
>
> Key: HBASE-19473
> URL: https://issues.apache.org/jira/browse/HBASE-19473
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19473.1.patch, HBASE-19473.2.patch,
> HBASE-19473.3.patch
>
>
> # Remove superfluous logging code guard
> # Simplify some of the code
> # Use ArrayDeque instead of LinkedList for queue implementation
> https://docs.oracle.com/javase/8/docs/api/index.html?java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19596) RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes
[ https://issues.apache.org/jira/browse/HBASE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310827#comment-16310827 ] Hudson commented on HBASE-19596: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4340 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4340/]) HBASE-19596 RegionMetrics/ServerMetrics/ClusterMetrics should apply to (chia7712: rev 8119acfca7e35cd7c4c203a397b970a50d7d7574) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestFavoredStochasticBalancerPickers.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailoverBalancerPersistence.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterStatusChore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/UnbalanceRegionsAction.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java * (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestClientClusterStatus.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionMetrics.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/MoveRegionsOfTableAction.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestStochasticBalancerJmxMetrics.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncDecommissionAdminApi.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/StripeCompactionsPerformanceEvaluation.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterOperationsForRegionReplicas.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseCluster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestInterfaceAlign.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestLazyCfLoading.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncClusterAdminApi.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRegionSizeCalculator.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionLoad.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * (edit)
[jira] [Commented] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310822#comment-16310822
]
Hudson commented on HBASE-18806:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4340 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4340/])
HBASE-18806 VerifyRep by snapshot need not to restore snapshot for each
(openinx: rev 6e136f26bf0761797716b532b1a8c4984bf80c58)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
* (edit)
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
* (edit)
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* (edit)
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
* (edit)
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/client/TableSnapshotScanner.java
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch,
> HBASE-18806.v3.patch, HBASE-18806.v3.patch, HBASE-18806.v4.patch,
> HBASE-18806.v4.patch, HBASE-18806.v5.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19613) Miscellaneous changes to WALSplitter
[ https://issues.apache.org/jira/browse/HBASE-19613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310823#comment-16310823 ] Hudson commented on HBASE-19613: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4340 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4340/]) HBASE-19613 Miscellaneous changes to WALSplitter. (appy: rev 301062566ac6e32d5bc3c6dbfd819b5e62742e8c) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java > Miscellaneous changes to WALSplitter > > > Key: HBASE-19613 > URL: https://issues.apache.org/jira/browse/HBASE-19613 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19613.1.patch, HBASE-19613.2.patch > > > * Use ArrayList instead LinkedList > * Use Apache Commons where appropriate > * Parameterize and improve logging -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310821#comment-16310821 ] Appy commented on HBASE-19483: -- Sorry, back again, first thing tomorrow morning. There were a lot of nice patches by new contributor, was giving extra love to them :) > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: 19483.master.011.patch, 19483.v11.patch, > 19483.v11.patch, HBASE-19483.master.001.patch, HBASE-19483.master.002.patch, > HBASE-19483.master.003.patch, HBASE-19483.master.004.patch, > HBASE-19483.master.005.patch, HBASE-19483.master.006.patch, > HBASE-19483.master.007.patch, HBASE-19483.master.008.patch, > HBASE-19483.master.009.patch, HBASE-19483.master.010.patch, > HBASE-19483.master.011.patch, HBASE-19483.master.011.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgroup / > get_rsgroup commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader
[
https://issues.apache.org/jira/browse/HBASE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved HBASE-15580.
Resolution: Fixed
Hadoop Flags: Reviewed
> Tag coprocessor limitedprivate scope to StoreFile.Reader
>
>
> Key: HBASE-15580
> URL: https://issues.apache.org/jira/browse/HBASE-15580
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 1.4.1, 1.5.0
>
> Attachments: HBASE-15580.patch, HBASE-15580_branch-1.0.patch
>
>
> For phoenix local indexing we need to have custom storefile reader
> constructor(IndexHalfStoreFileReader) to distinguish from other storefile
> readers. So wanted to mark StoreFile.Reader scope as
> InterfaceAudience.LimitedPrivate("Coprocessor")
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19636) All rs should already start work with the new peer change when replication peer procedure is finished
[
https://issues.apache.org/jira/browse/HBASE-19636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310817#comment-16310817
]
Hadoop QA commented on HBASE-19636:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19397 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} HBASE-19397 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hbase-replication: The patch generated 0 new + 20
unchanged - 1 fixed = 20 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
4s{color} | {color:red} hbase-server: The patch generated 2 new + 29 unchanged
- 3 fixed = 31 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 5s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.TestReplicationDisableInactivePeer |
| | hadoop.hbase.replication.regionserver.TestReplicationSourceManagerZkImpl |
| | hadoop.hbase.replication.TestReplicationDroppedTables |
| | hadoop.hbase.replication.TestPerTableCFReplication |
| | hadoop.hbase.replication.TestNamespaceReplication |
| | hadoop.hbase.replication.TestMasterReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19636 |
| JIRA Patch URL |
[jira] [Updated] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader
[
https://issues.apache.org/jira/browse/HBASE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-15580:
---
Fix Version/s: (was: 1.2.7)
(was: 1.3.2)
> Tag coprocessor limitedprivate scope to StoreFile.Reader
>
>
> Key: HBASE-15580
> URL: https://issues.apache.org/jira/browse/HBASE-15580
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 1.4.1, 1.5.0
>
> Attachments: HBASE-15580.patch, HBASE-15580_branch-1.0.patch
>
>
> For phoenix local indexing we need to have custom storefile reader
> constructor(IndexHalfStoreFileReader) to distinguish from other storefile
> readers. So wanted to mark StoreFile.Reader scope as
> InterfaceAudience.LimitedPrivate("Coprocessor")
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader
[
https://issues.apache.org/jira/browse/HBASE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310808#comment-16310808
]
Chia-Ping Tsai commented on HBASE-15580:
HBASE-16225 had made some APIs changes for StoreFile.Reader in branch-1.4+ so I
won't commit the patch to branch-1.3 and branch-1.2 in order to keep our
compatibility rule.
> Tag coprocessor limitedprivate scope to StoreFile.Reader
>
>
> Key: HBASE-15580
> URL: https://issues.apache.org/jira/browse/HBASE-15580
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 1.4.1, 1.5.0
>
> Attachments: HBASE-15580.patch, HBASE-15580_branch-1.0.patch
>
>
> For phoenix local indexing we need to have custom storefile reader
> constructor(IndexHalfStoreFileReader) to distinguish from other storefile
> readers. So wanted to mark StoreFile.Reader scope as
> InterfaceAudience.LimitedPrivate("Coprocessor")
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19703) Functionality added as part of HBASE-12583 is not working after moving the split code to master
[
https://issues.apache.org/jira/browse/HBASE-19703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310800#comment-16310800
]
Rajeshbabu Chintaguntla commented on HBASE-19703:
-
Since we are not able to access HRegion cannot use the split policy in the
SplitTableRegionProcedure so one possible solution is that we can use Column
Family level property to set whether to skip the HFile key range check with
split key.
WDYT [~stack][~anoop.hbase][~ramkrishna.s.vasude...@gmail.com]? If it's fine
will make a patch accordingly.
> Functionality added as part of HBASE-12583 is not working after moving the
> split code to master
> ---
>
> Key: HBASE-19703
> URL: https://issues.apache.org/jira/browse/HBASE-19703
> Project: HBase
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0-beta-2
>
>
> As part of HBASE-12583 we are passing split policy to
> HRegionFileSystem#splitStoreFile so that we can allow to create reference
> files even the split key is out of HFile key range. This is needed for Local
> Indexing implementation in Phoenix. But now after moving the split code to
> master just passing null for split policy.
> {noformat}
> final String familyName = Bytes.toString(family);
> final Path path_first =
> regionFs.splitStoreFile(this.daughter_1_RI, familyName, sf, splitRow,
> false, null);
> final Path path_second =
> regionFs.splitStoreFile(this.daughter_2_RI, familyName, sf, splitRow,
> true, null);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18452:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
> --
>
> Key: HBASE-18452
> URL: https://issues.apache.org/jira/browse/HBASE-18452
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18452.v1.patch, HBASE-18452.v2.patch,
> HBASE-18452.v2.patch
>
>
> I've ported HBASE-16466 to our internal hbase branch, and tested the
> feature under our kerberos cluster.
> The problem we encountered is:
> {code}
> 17/07/25 21:21:23 INFO mapreduce.Job: Task Id :
> attempt_1500987232138_0004_m_03_2, Status : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "hadoop-yarn-host";
> destination host is: "hadoop-namenode-host":15200;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1408)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2029)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1195)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1207)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:778)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:769)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:748)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5125)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:60)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:191)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:148)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Commented] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310798#comment-16310798
]
Zheng Hu commented on HBASE-18452:
--
Pushed to master & branch-2 , Thanks [~zghaobac] for reviewing.
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
> --
>
> Key: HBASE-18452
> URL: https://issues.apache.org/jira/browse/HBASE-18452
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18452.v1.patch, HBASE-18452.v2.patch,
> HBASE-18452.v2.patch
>
>
> I've ported HBASE-16466 to our internal hbase branch, and tested the
> feature under our kerberos cluster.
> The problem we encountered is:
> {code}
> 17/07/25 21:21:23 INFO mapreduce.Job: Task Id :
> attempt_1500987232138_0004_m_03_2, Status : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "hadoop-yarn-host";
> destination host is: "hadoop-namenode-host":15200;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1408)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2029)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1195)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1207)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:778)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:769)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:748)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5125)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:60)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:191)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:148)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Updated] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader
[
https://issues.apache.org/jira/browse/HBASE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-15580:
---
Fix Version/s: (was: 3.0.0)
(was: 2.0.0)
1.2.7
1.4.1
1.3.2
> Tag coprocessor limitedprivate scope to StoreFile.Reader
>
>
> Key: HBASE-15580
> URL: https://issues.apache.org/jira/browse/HBASE-15580
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7
>
> Attachments: HBASE-15580.patch, HBASE-15580_branch-1.0.patch
>
>
> For phoenix local indexing we need to have custom storefile reader
> constructor(IndexHalfStoreFileReader) to distinguish from other storefile
> readers. So wanted to mark StoreFile.Reader scope as
> InterfaceAudience.LimitedPrivate("Coprocessor")
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader
[
https://issues.apache.org/jira/browse/HBASE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310779#comment-16310779
]
Chia-Ping Tsai commented on HBASE-15580:
HBASE-18446 had tagged the LP scope to StoreFileReader so it is time to make
this in branch-1.
> Tag coprocessor limitedprivate scope to StoreFile.Reader
>
>
> Key: HBASE-15580
> URL: https://issues.apache.org/jira/browse/HBASE-15580
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7
>
> Attachments: HBASE-15580.patch, HBASE-15580_branch-1.0.patch
>
>
> For phoenix local indexing we need to have custom storefile reader
> constructor(IndexHalfStoreFileReader) to distinguish from other storefile
> readers. So wanted to mark StoreFile.Reader scope as
> InterfaceAudience.LimitedPrivate("Coprocessor")
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19704) region split happened when it contains reference file
chenxu created HBASE-19704: -- Summary: region split happened when it contains reference file Key: HBASE-19704 URL: https://issues.apache.org/jira/browse/HBASE-19704 Project: HBase Issue Type: Bug Reporter: chenxu In our product cluster, some Region’s dir contains reference file, but it also splited, that's violate ConstantSizeRegionSplitPolicy.shouldSplit() method, and CatalogJanitor can’t collect it. I suspect it was due to HBASE-13082 which we recently introduced, after introduce this feature, the comapcted files were removed asynchronously. when region split happens and HStore.close() execute, if compactedfiles can’t satisfy (r != null && r.isCompactedAway() && !r.isReferencedInReads()) it will be always exist. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19596) RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes
[ https://issues.apache.org/jira/browse/HBASE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19596: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Commit the patch with [~zghaobac]'s suggestions. > RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes > - > > Key: HBASE-19596 > URL: https://issues.apache.org/jira/browse/HBASE-19596 > Project: HBase > Issue Type: Task > Components: Client >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19596.v0.patch, HBASE-19596.v1.patch, > HBASE-19596.v2.patch, HBASE-19596.v3.patch > > > ServerLoad/RegionLoad/ClusterLoad are deprecated now. This issue will > deprecate all related methods in our Public classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19703) Functionality added as part of HBASE-12583 is not working after moving the split code to master
[
https://issues.apache.org/jira/browse/HBASE-19703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310723#comment-16310723
]
Rajeshbabu Chintaguntla commented on HBASE-19703:
-
Ping [~stack] [~anoop.hbase] [~ramkrishna.s.vasude...@gmail.com].
> Functionality added as part of HBASE-12583 is not working after moving the
> split code to master
> ---
>
> Key: HBASE-19703
> URL: https://issues.apache.org/jira/browse/HBASE-19703
> Project: HBase
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0-beta-2
>
>
> As part of HBASE-12583 we are passing split policy to
> HRegionFileSystem#splitStoreFile so that we can allow to create reference
> files even the split key is out of HFile key range. This is needed for Local
> Indexing implementation in Phoenix. But now after moving the split code to
> master just passing null for split policy.
> {noformat}
> final String familyName = Bytes.toString(family);
> final Path path_first =
> regionFs.splitStoreFile(this.daughter_1_RI, familyName, sf, splitRow,
> false, null);
> final Path path_second =
> regionFs.splitStoreFile(this.daughter_2_RI, familyName, sf, splitRow,
> true, null);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19703) Functionality added as part of HBASE-12583 is not working after moving the split code to master
Rajeshbabu Chintaguntla created HBASE-19703:
---
Summary: Functionality added as part of HBASE-12583 is not working
after moving the split code to master
Key: HBASE-19703
URL: https://issues.apache.org/jira/browse/HBASE-19703
Project: HBase
Issue Type: Bug
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Fix For: 2.0.0-beta-2
As part of HBASE-12583 we are passing split policy to
HRegionFileSystem#splitStoreFile so that we can allow to create reference files
even the split key is out of HFile key range. This is needed for Local Indexing
implementation in Phoenix. But now after moving the split code to master just
passing null for split policy.
{noformat}
final String familyName = Bytes.toString(family);
final Path path_first =
regionFs.splitStoreFile(this.daughter_1_RI, familyName, sf, splitRow,
false, null);
final Path path_second =
regionFs.splitStoreFile(this.daughter_2_RI, familyName, sf, splitRow,
true, null);
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310715#comment-16310715
]
Hadoop QA commented on HBASE-18452:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 8m
5s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
49s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
53s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-18452 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904513/HBASE-18452.v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8cdffd45f14e 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4ba741674d |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10878/testReport/ |
| modules | C: hbase-mapreduce U: hbase-mapreduce |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10878/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
>
[jira] [Commented] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310714#comment-16310714
]
Hadoop QA commented on HBASE-18452:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
3s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
8s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-18452 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904515/HBASE-18452.v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4c543ad736d1 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 4ba741674d |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10880/testReport/ |
| modules | C: hbase-mapreduce U: hbase-mapreduce |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10880/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[
https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310713#comment-16310713
]
BELUGA BEHR commented on HBASE-19651:
-
@stack Actually, the only change in Guava is that LimitedInputStream was moved
to a private class and ByteStream.limit simply instantiates it for you. So in
essence, HBase will still be using the LimitedInputStream:
https://github.com/google/guava/blob/fd919e54a55ba169dc7d9f54b7b3485aa7fa0970/guava/src/com/google/common/io/ByteStreams.java#L629
I think I blew away the tests, but all that needed to be tested is the overhead
imposed by the implementation checking if the stream should be closed.
{code}
InputStream is = new ByteArrayInputStream(1024 * 1024 * 256); // 256MB
// InputStream lis = ByteStream.limit(is, 1024 * 1024 * 256);
// InputStream lis = new BoundedInputStream(is, 1024 * 1024 * 256);
byte[] buf = new byte[256]; // intentionally small buffer for lots of reads
while (-1 != lis.read(buf)); // time this action
{code}
> Remove LimitInputStream
> ---
>
> Key: HBASE-19651
> URL: https://issues.apache.org/jira/browse/HBASE-19651
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch,
> HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch,
> HBASE-19651.6.patch
>
>
> Let us "drink our own champagne" and use the existing Apache Commons
> BoundedInputStream instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19702) Improve RSGroupInfo constructor
[ https://issues.apache.org/jira/browse/HBASE-19702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310712#comment-16310712 ] Chia-Ping Tsai commented on HBASE-19702: bq. Maybe it can not be null? Or can it be? RSGroupInfo is a pojo so the null should be acceptable I think. bq. I feel like we should start using @NotNull in our codebase. I can't agree with you more.:) > Improve RSGroupInfo constructor > --- > > Key: HBASE-19702 > URL: https://issues.apache.org/jira/browse/HBASE-19702 > Project: HBase > Issue Type: Improvement >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19702) Improve RSGroupInfo constructor
[ https://issues.apache.org/jira/browse/HBASE-19702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310709#comment-16310709 ] Appy commented on HBASE-19702: -- bq. It leads to NullPointerException if tables is null Maybe it can not be null? Or can it be? I feel like we should start using @NotNull in our codebase. > Improve RSGroupInfo constructor > --- > > Key: HBASE-19702 > URL: https://issues.apache.org/jira/browse/HBASE-19702 > Project: HBase > Issue Type: Improvement >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19473) Miscellaneous changes to ClientScanner
[
https://issues.apache.org/jira/browse/HBASE-19473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-19473:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Miscellaneous changes to ClientScanner
> --
>
> Key: HBASE-19473
> URL: https://issues.apache.org/jira/browse/HBASE-19473
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19473.1.patch, HBASE-19473.2.patch,
> HBASE-19473.3.patch
>
>
> # Remove superfluous logging code guard
> # Simplify some of the code
> # Use ArrayDeque instead of LinkedList for queue implementation
> https://docs.oracle.com/javase/8/docs/api/index.html?java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19473) Miscellaneous changes to ClientScanner
[
https://issues.apache.org/jira/browse/HBASE-19473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-19473:
-
Summary: Miscellaneous changes to ClientScanner (was: Review of
ClientScanner Class)
> Miscellaneous changes to ClientScanner
> --
>
> Key: HBASE-19473
> URL: https://issues.apache.org/jira/browse/HBASE-19473
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19473.1.patch, HBASE-19473.2.patch,
> HBASE-19473.3.patch
>
>
> # Remove superfluous logging code guard
> # Simplify some of the code
> # Use ArrayDeque instead of LinkedList for queue implementation
> https://docs.oracle.com/javase/8/docs/api/index.html?java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19613) Miscellaneous changes to WALSplitter
[ https://issues.apache.org/jira/browse/HBASE-19613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19613: - Summary: Miscellaneous changes to WALSplitter (was: Review of WALSplitter Class) > Miscellaneous changes to WALSplitter > > > Key: HBASE-19613 > URL: https://issues.apache.org/jira/browse/HBASE-19613 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19613.1.patch, HBASE-19613.2.patch > > > * Use ArrayList instead LinkedList > * Use Apache Commons where appropriate > * Parameterize and improve logging -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19473) Review of ClientScanner Class
[
https://issues.apache.org/jira/browse/HBASE-19473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-19473:
-
Fix Version/s: 2.0.0-beta-2
> Review of ClientScanner Class
> -
>
> Key: HBASE-19473
> URL: https://issues.apache.org/jira/browse/HBASE-19473
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19473.1.patch, HBASE-19473.2.patch,
> HBASE-19473.3.patch
>
>
> # Remove superfluous logging code guard
> # Simplify some of the code
> # Use ArrayDeque instead of LinkedList for queue implementation
> https://docs.oracle.com/javase/8/docs/api/index.html?java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19691) Do not require ADMIN permission for obtaining ClusterStatus
[ https://issues.apache.org/jira/browse/HBASE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310704#comment-16310704 ] Hudson commented on HBASE-19691: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4339 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4339/]) HBASE-19691 Removes Global(A) requirement for getClusterStatus (elserj: rev 9a98bb4ce9d3e600a2b982995914222c305ebe8a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) src/main/asciidoc/_chapters/appendix_acl_matrix.adoc > Do not require ADMIN permission for obtaining ClusterStatus > --- > > Key: HBASE-19691 > URL: https://issues.apache.org/jira/browse/HBASE-19691 > Project: HBase > Issue Type: Bug >Reporter: Romil Choksi >Assignee: Josh Elser >Priority: Critical > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: HBASE-19691.001.branch-2.patch > > > Appears to be a regression introduced by HBASE-19131. Operations that attempt > to obtain the `status` from the HMaster now fail if the requesting user > doesn't have global ADMIN permission. > Discussion: > https://lists.apache.org/thread.html/f1cd2a50e5c460879c97043790b33aa375cd6b217455d611c3417e3d@%3Cdev.hbase.apache.org%3E > Thanks to Romil for letting us know about this one. > FYI [~stack] [~chia7712]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19596) RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes
[ https://issues.apache.org/jira/browse/HBASE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310696#comment-16310696 ] Guanghao Zhang commented on HBASE-19596: bq. @return a region load list of all regions hosted on a region server These need be updated to region metrics (Admin.java). +1 for else. You can fix this when commit. :-) > RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes > - > > Key: HBASE-19596 > URL: https://issues.apache.org/jira/browse/HBASE-19596 > Project: HBase > Issue Type: Task > Components: Client >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19596.v0.patch, HBASE-19596.v1.patch, > HBASE-19596.v2.patch, HBASE-19596.v3.patch > > > ServerLoad/RegionLoad/ClusterLoad are deprecated now. This issue will > deprecate all related methods in our Public classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310692#comment-16310692
]
Guanghao Zhang commented on HBASE-18452:
+1
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
> --
>
> Key: HBASE-18452
> URL: https://issues.apache.org/jira/browse/HBASE-18452
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18452.v1.patch, HBASE-18452.v2.patch,
> HBASE-18452.v2.patch
>
>
> I've ported HBASE-16466 to our internal hbase branch, and tested the
> feature under our kerberos cluster.
> The problem we encountered is:
> {code}
> 17/07/25 21:21:23 INFO mapreduce.Job: Task Id :
> attempt_1500987232138_0004_m_03_2, Status : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "hadoop-yarn-host";
> destination host is: "hadoop-namenode-host":15200;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1408)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2029)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1195)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1207)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:778)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:769)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:748)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5125)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:60)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:191)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:148)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Updated] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18452:
-
Fix Version/s: 2.0.0-beta-2
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
> --
>
> Key: HBASE-18452
> URL: https://issues.apache.org/jira/browse/HBASE-18452
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18452.v1.patch, HBASE-18452.v2.patch,
> HBASE-18452.v2.patch
>
>
> I've ported HBASE-16466 to our internal hbase branch, and tested the
> feature under our kerberos cluster.
> The problem we encountered is:
> {code}
> 17/07/25 21:21:23 INFO mapreduce.Job: Task Id :
> attempt_1500987232138_0004_m_03_2, Status : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "hadoop-yarn-host";
> destination host is: "hadoop-namenode-host":15200;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1408)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2029)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1195)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1207)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:778)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:769)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:748)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5125)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:60)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:191)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:148)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
>
[jira] [Updated] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18452:
-
Affects Version/s: 2.0.0-beta-2
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
> --
>
> Key: HBASE-18452
> URL: https://issues.apache.org/jira/browse/HBASE-18452
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18452.v1.patch, HBASE-18452.v2.patch,
> HBASE-18452.v2.patch
>
>
> I've ported HBASE-16466 to our internal hbase branch, and tested the
> feature under our kerberos cluster.
> The problem we encountered is:
> {code}
> 17/07/25 21:21:23 INFO mapreduce.Job: Task Id :
> attempt_1500987232138_0004_m_03_2, Status : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "hadoop-yarn-host";
> destination host is: "hadoop-namenode-host":15200;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1408)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2029)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1195)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1207)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:778)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:769)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:748)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5125)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:60)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:191)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:148)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
>
[jira] [Commented] (HBASE-19702) Improve RSGroupInfo constructor
[ https://issues.apache.org/jira/browse/HBASE-19702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310689#comment-16310689 ] Chia-Ping Tsai commented on HBASE-19702: bq. When servers is not null, addAll(servers) tries to add all items in servers again. Seems not needed nice catch. bq. new TreeSet<>(tables) has no null check on tables. The constructor of TreeSet does not do the null check either. It leads to NullPointerException if tables is null agreed. > Improve RSGroupInfo constructor > --- > > Key: HBASE-19702 > URL: https://issues.apache.org/jira/browse/HBASE-19702 > Project: HBase > Issue Type: Improvement >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19702) Improve RSGroupInfo constructor
[ https://issues.apache.org/jira/browse/HBASE-19702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19702: --- Fix Version/s: 2.0.0-beta-2 > Improve RSGroupInfo constructor > --- > > Key: HBASE-19702 > URL: https://issues.apache.org/jira/browse/HBASE-19702 > Project: HBase > Issue Type: Improvement >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18452:
-
Attachment: HBASE-18452.v2.patch
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
> --
>
> Key: HBASE-18452
> URL: https://issues.apache.org/jira/browse/HBASE-18452
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-18452.v1.patch, HBASE-18452.v2.patch,
> HBASE-18452.v2.patch
>
>
> I've ported HBASE-16466 to our internal hbase branch, and tested the
> feature under our kerberos cluster.
> The problem we encountered is:
> {code}
> 17/07/25 21:21:23 INFO mapreduce.Job: Task Id :
> attempt_1500987232138_0004_m_03_2, Status : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "hadoop-yarn-host";
> destination host is: "hadoop-namenode-host":15200;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1408)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2029)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1195)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1207)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:778)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:769)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:748)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5125)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:60)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:191)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:148)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at
>
[jira] [Commented] (HBASE-19473) Review of ClientScanner Class
[
https://issues.apache.org/jira/browse/HBASE-19473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310688#comment-16310688
]
Appy commented on HBASE-19473:
--
Liked the flow changes. Others lgtm.
Pushing to master and branch-2.
Thanks for the patch [~belugabehr].
> Review of ClientScanner Class
> -
>
> Key: HBASE-19473
> URL: https://issues.apache.org/jira/browse/HBASE-19473
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19473.1.patch, HBASE-19473.2.patch,
> HBASE-19473.3.patch
>
>
> # Remove superfluous logging code guard
> # Simplify some of the code
> # Use ArrayDeque instead of LinkedList for queue implementation
> https://docs.oracle.com/javase/8/docs/api/index.html?java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19636) All rs should already start work with the new peer change when replication peer procedure is finished
[ https://issues.apache.org/jira/browse/HBASE-19636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310685#comment-16310685 ] Guanghao Zhang commented on HBASE-19636: Make the Synchronization specification clear in ReplicationSourceManager. Now only have two things need synchronization. For normal replication source, need synchronized on latestPaths to avoid the new open source miss new log. For recovered sources, need synchronized on oldSources to avoid adding recovered source for the to-be-removed peer. > All rs should already start work with the new peer change when replication > peer procedure is finished > - > > Key: HBASE-19636 > URL: https://issues.apache.org/jira/browse/HBASE-19636 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-19636.HBASE-19397.001.patch, > HBASE-19636.HBASE-19397.002.patch > > > When replication peer operations use zk, the master will modify zk directly. > Then the rs will asynchronous track the zk event to start work with the new > peer change. When replication peer operations use procedure, need to make > sure this process is synchronous. All rs should already start work with the > new peer change when procedure is finished. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19651: -- Release Note: HBase had copied from guava the file LmiitedInputStream. This commit removes the copied file in favor of (our internal, shaded) guava's ByteStreams.limit. Guava 14.0's LIS noted: "Use ByteStreams.limit(java.io.InputStream, long) instead. This class is scheduled to be removed in Guava release 15.0." > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18452) VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env.
[
https://issues.apache.org/jira/browse/HBASE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18452:
-
Attachment: HBASE-18452.v2.patch
Rebase
> VerifyReplication by Snapshot should cache HDFS token before submit job for
> kerberos env.
> --
>
> Key: HBASE-18452
> URL: https://issues.apache.org/jira/browse/HBASE-18452
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-18452.v1.patch, HBASE-18452.v2.patch
>
>
> I've ported HBASE-16466 to our internal hbase branch, and tested the
> feature under our kerberos cluster.
> The problem we encountered is:
> {code}
> 17/07/25 21:21:23 INFO mapreduce.Job: Task Id :
> attempt_1500987232138_0004_m_03_2, Status : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "hadoop-yarn-host";
> destination host is: "hadoop-namenode-host":15200;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1408)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2029)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1195)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1207)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.checkRegionInfoOnFilesystem(HRegionFileSystem.java:778)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:769)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:748)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5125)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:60)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:191)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:148)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1885)
> at
>
[jira] [Updated] (HBASE-19636) All rs should already start work with the new peer change when replication peer procedure is finished
[ https://issues.apache.org/jira/browse/HBASE-19636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19636: --- Attachment: HBASE-19636.HBASE-19397.002.patch > All rs should already start work with the new peer change when replication > peer procedure is finished > - > > Key: HBASE-19636 > URL: https://issues.apache.org/jira/browse/HBASE-19636 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-19636.HBASE-19397.001.patch, > HBASE-19636.HBASE-19397.002.patch > > > When replication peer operations use zk, the master will modify zk directly. > Then the rs will asynchronous track the zk event to start work with the new > peer change. When replication peer operations use procedure, need to make > sure this process is synchronous. All rs should already start work with the > new peer change when procedure is finished. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19702) Improve RSGroupInfo constructor
[
https://issues.apache.org/jira/browse/HBASE-19702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310676#comment-16310676
]
Xiang Li edited comment on HBASE-19702 at 1/4/18 3:49 AM:
--
{code:title=hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java|borderStyle=solid}
RSGroupInfo(String name, SortedSet servers, SortedSet
tables) {
this.name = name;
this.servers = servers == null? new TreeSet<>(): servers;
this.servers.addAll(servers);
this.tables = new TreeSet<>(tables);
}
{code}
2 improvements could be made:
* When servers is not null, addAll(servers) tries to add all items in servers
again. Seems not needed
* new TreeSet<>(tables) has no null check on tables. The constructor of TreeSet
does not do the null check either. It leads to NullPointerException if tables
is null
was (Author: water):
{code:title=hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java|borderStyle=solid}
RSGroupInfo(String name, SortedSet servers, SortedSet
tables) {
this.name = name;
this.servers = servers == null? new TreeSet<>(): servers;
this.servers.addAll(servers);
this.tables = new TreeSet<>(tables);
}
{code}
2 improvements could be made:
* When servers is not null, addAll(servers) tries to add all items in servers
again. Seems not needed
* new TreeSet<>(tables) has not null check. The constructor of TreeSet does not
do the null check either. It leads to NullPointerException if tables is null
> Improve RSGroupInfo constructor
> ---
>
> Key: HBASE-19702
> URL: https://issues.apache.org/jira/browse/HBASE-19702
> Project: HBase
> Issue Type: Improvement
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310682#comment-16310682 ] stack commented on HBASE-19651: --- +1 from me. [~belugabehr] did you compare ByteStreams to LimitedInputStream? Also, your test, no matter how hacky, mind pasting the code you used for the archeologists who might be trying to follow along a few years on? I stuck a bit of a release note here. Edit it if it is not correct. This is great stuff [~belugabehr] Thanks for cleanup. > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-19702) Improve RSGroupInfo constructor
[ https://issues.apache.org/jira/browse/HBASE-19702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Li reassigned HBASE-19702: Assignee: Xiang Li > Improve RSGroupInfo constructor > --- > > Key: HBASE-19702 > URL: https://issues.apache.org/jira/browse/HBASE-19702 > Project: HBase > Issue Type: Improvement >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19588) Additional jar dependencies needed for mapreduce PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19588:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0-beta-2
1.4.1
Status: Resolved (was: Patch Available)
Pushed to branch-1.4+ Thanks for the fix [~chu11] (thanks for review
[~chia7712])
> Additional jar dependencies needed for mapreduce PerformanceEvaluation
> --
>
> Key: HBASE-19588
> URL: https://issues.apache.org/jira/browse/HBASE-19588
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.4.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Minor
> Fix For: 1.4.1, 2.0.0-beta-2
>
> Attachments: HBASE-19588.branch-1.4.patch
>
>
> I have a unit test that runs a simple PerformanceEvaluation test to make sure
> things are basically working
> {noformat}
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=5
> sequentialWrite 1
> {noformat}
> This test runs against Hadoop 2.7.0 and works against all past versions
> 0.99.0 and up. It broke with 1.4.0 with the following error.
> {noformat}
> 2017-12-21 13:49:40,974 INFO [main] mapreduce.Job: Task Id :
> attempt_1513892752187_0002_m_04_2, Status : FAILED
> Error: java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:297)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:250)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> ... 12 more
> Caused by: java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeper.(MetricsZooKeeper.java:38)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:130)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:143)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:181)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:155)
> at
> org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.(ZooKeeperKeepAliveConnection.java:43)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1737)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:945)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:721)
> ... 17 more
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:224)
> at
[jira] [Commented] (HBASE-19702) Improve RSGroupInfo constructor
[
https://issues.apache.org/jira/browse/HBASE-19702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310676#comment-16310676
]
Xiang Li commented on HBASE-19702:
--
{code:title=hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java|borderStyle=solid}
RSGroupInfo(String name, SortedSet servers, SortedSet
tables) {
this.name = name;
this.servers = servers == null? new TreeSet<>(): servers;
this.servers.addAll(servers);
this.tables = new TreeSet<>(tables);
}
{code}
2 improvements could be made:
* When servers is not null, addAll(servers) tries to add all items in servers
again. Seems not needed
* new TreeSet<>(tables) has not null check. The constructor of TreeSet does not
do the null check either. It leads to NullPointerException if tables is null
> Improve RSGroupInfo constructor
> ---
>
> Key: HBASE-19702
> URL: https://issues.apache.org/jira/browse/HBASE-19702
> Project: HBase
> Issue Type: Improvement
>Reporter: Xiang Li
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18059) The scanner order for memstore scanners are wrong
[
https://issues.apache.org/jira/browse/HBASE-18059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310674#comment-16310674
]
Jingyun Tian commented on HBASE-18059:
--
As [~Appy] said before, the only situation when the order is determined by
getScannerOrder() is SF vs SF and both have no seqId.
For SegmentScanner vs SegmentScanner, since all cells in memstore have SeqId,
so it's impossible to reach getScannerOrder().
I think it's better to modify the comment of getScannerOrder() to:
{code}
/**
* Get the order of this KeyValueScanner. This is only relevant for
StoreFileScanners.
* This is required for comparing multiple files to find out which one has
the latest
* data. StoreFileScanners are ordered from 0 (oldest) to newest in
increasing order.
*/
{code}
And I think for CompactingMemStore and DefaultMemStore:
{code}
long order = 1 + pipelineList.size() + snapshotList.size();
{code}
Although this part doesn't work, counting order from LONG.MAX makes more sense.
[~Apache9]
> The scanner order for memstore scanners are wrong
> -
>
> Key: HBASE-18059
> URL: https://issues.apache.org/jira/browse/HBASE-18059
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan, Scanners
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Jingyun Tian
>Priority: Critical
> Fix For: 2.0.0
>
>
> This is comments for KeyValueScanner.getScannerOrder
> {code:title=KeyValueScanner.java}
> /**
>* Get the order of this KeyValueScanner. This is only relevant for
> StoreFileScanners and
>* MemStoreScanners (other scanners simply return 0). This is required for
> comparing multiple
>* files to find out which one has the latest data. StoreFileScanners are
> ordered from 0
>* (oldest) to newest in increasing order. MemStoreScanner gets LONG.max
> since it always
>* contains freshest data.
>*/
> long getScannerOrder();
> {code}
> As now we may have multiple memstore scanners, I think the right way to
> select scanner order for memstore scanner is to ordered from Long.MAX_VALUE
> in decreasing order.
> But in CompactingMemStore and DefaultMemStore, the scanner order for memstore
> scanner is also start from 0, which will be messed up with StoreFileScanners.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[
https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310673#comment-16310673
]
stack commented on HBASE-19694:
---
I can take this in a day or two np [~Apache9]
> The initialization order for a fresh cluster is incorrect
> -
>
> Key: HBASE-19694
> URL: https://issues.apache.org/jira/browse/HBASE-19694
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> The cluster id will set once we become the active master in
> finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and
> finishActiveMasterInitialization are both called in a thread to make the
> constructor of HMaster return without blocking. And since HMaster itself is
> also a HRegionServer, it will create a Connection and then start calling
> reportForDuty. And when creating the ConnectionImplementation, we will read
> the cluster id from zk, but the cluster id may have not been set yet since it
> is set in another thread, we will get an exception and use the default
> cluster id instead.
> I always get this when running UTs which will start a mini cluster
> {noformat}
> 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848]
> client.ConnectionImplementation(528): Retrieve cluster id failed
> java.util.concurrent.ExecutionException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hbase/hbaseid
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/hbaseid
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163)
> at
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[
https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19694:
--
Fix Version/s: 2.0.0-beta-2
> The initialization order for a fresh cluster is incorrect
> -
>
> Key: HBASE-19694
> URL: https://issues.apache.org/jira/browse/HBASE-19694
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> The cluster id will set once we become the active master in
> finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and
> finishActiveMasterInitialization are both called in a thread to make the
> constructor of HMaster return without blocking. And since HMaster itself is
> also a HRegionServer, it will create a Connection and then start calling
> reportForDuty. And when creating the ConnectionImplementation, we will read
> the cluster id from zk, but the cluster id may have not been set yet since it
> is set in another thread, we will get an exception and use the default
> cluster id instead.
> I always get this when running UTs which will start a mini cluster
> {noformat}
> 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848]
> client.ConnectionImplementation(528): Retrieve cluster id failed
> java.util.concurrent.ExecutionException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hbase/hbaseid
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/hbaseid
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163)
> at
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[
https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19694:
--
Priority: Critical (was: Major)
> The initialization order for a fresh cluster is incorrect
> -
>
> Key: HBASE-19694
> URL: https://issues.apache.org/jira/browse/HBASE-19694
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> The cluster id will set once we become the active master in
> finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and
> finishActiveMasterInitialization are both called in a thread to make the
> constructor of HMaster return without blocking. And since HMaster itself is
> also a HRegionServer, it will create a Connection and then start calling
> reportForDuty. And when creating the ConnectionImplementation, we will read
> the cluster id from zk, but the cluster id may have not been set yet since it
> is set in another thread, we will get an exception and use the default
> cluster id instead.
> I always get this when running UTs which will start a mini cluster
> {noformat}
> 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848]
> client.ConnectionImplementation(528): Retrieve cluster id failed
> java.util.concurrent.ExecutionException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hbase/hbaseid
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/hbaseid
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163)
> at
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19702) Improve RSGroupInfo constructor
Xiang Li created HBASE-19702: Summary: Improve RSGroupInfo constructor Key: HBASE-19702 URL: https://issues.apache.org/jira/browse/HBASE-19702 Project: HBase Issue Type: Improvement Reporter: Xiang Li Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19596) RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes
[
https://issues.apache.org/jira/browse/HBASE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310671#comment-16310671
]
Hadoop QA commented on HBASE-19596:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 52 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} hbase-client: The patch generated 0 new + 461
unchanged - 13 fixed = 461 total (was 474) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} hbase-server: The patch generated 0 new + 1242
unchanged - 185 fixed = 1242 total (was 1427) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} hbase-mapreduce: The patch generated 0 new + 1
unchanged - 3 fixed = 1 total (was 4) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-rsgroup passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hbase-it: The patch generated 0 new + 84 unchanged -
8 fixed = 84 total (was 92) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hbase-rest passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 1s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
42s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}175m
1s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
31s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310666#comment-16310666 ] Appy commented on HBASE-17436: -- bq. I will move the json part to the rest endpoint. What should we do with the ui-part of the patch? I think it is a good addition to the UI if exposing the Regionstate enum values is not a problem... Also if we do not want to expose our enum values, should we do some mapping to other values which are "public concepts" just to make users able to get some insight into the state of the regions? Definitely a good addition to UI! And given dev discussion, and keeping operators' best interest in mind, let's expose the states as plain strings ;) And put a note in hbase book what the value means (ref to ascii doc of the class) and explicitly stating that it's private class and since enum values can change anytime, string values can too. Go for it, make it rest api. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19121) HBCK for AMv2
[ https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310665#comment-16310665 ] stack commented on HBASE-19121: --- Here is a case we need tooling for: region is reporting the below in the Master: 2017-12-31 10:12:52,611 WARN [ProcExecTimeout] assignment.AssignmentManager: TODO Handle stuck in transition: rit=OPENING, location=node8.com,16020,151469206, table=work_proposed, region=5b4b9a7b4e58da39a2072fdcb512df2f We should NEVER arrive at the above state. Up to this we've been trying to plug all the holes in AMv2 that might provoke the above but it is unlikely we'll see all failure scenarios before we ship 2.0.0 so there will be cases where regions will end up in the unassignable state (and operator will see logs like those above with associated regions offline). An hbck2 would intervene and if the region unassigned ten minutes or more, it'd move region back to an assignable state via Master invocations. See the thread [VOTE] The first hbase-2.0.0-beta-1 Release Candidate is available" for more. See [~jmspaggi] notes. > HBCK for AMv2 > - > > Key: HBASE-19121 > URL: https://issues.apache.org/jira/browse/HBASE-19121 > Project: HBase > Issue Type: Bug > Components: hbck >Reporter: stack > Fix For: 2.0.0-beta-2 > > > We don't have an hbck for the new AM. Old hbck may actually do damage going > against AMv2. > Fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19699) Revisit class ReplicationSourceManager
[ https://issues.apache.org/jira/browse/HBASE-19699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-19699: - Issue Type: Improvement (was: Sub-task) Parent: (was: HBASE-19397) > Revisit class ReplicationSourceManager > -- > > Key: HBASE-19699 > URL: https://issues.apache.org/jira/browse/HBASE-19699 > Project: HBase > Issue Type: Improvement > Components: proc-v2, Replication >Affects Versions: HBASE-19397 >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: HBASE-19397 > > > Code cleanup and simplification for ReplicationSourceManager. > Will start this work after HBASE-19636 resolved. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19490) Rare failure in TestRateLimiter
[ https://issues.apache.org/jira/browse/HBASE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310659#comment-16310659 ] Hudson commented on HBASE-19490: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #328 (See [https://builds.apache.org/job/HBase-1.3-IT/328/]) HBASE-19490 Rare failure in TestRateLimiter (chia7712: rev aded33e20609926e51ad8a1e3059e1364bae6ff3) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java > Rare failure in TestRateLimiter > --- > > Key: HBASE-19490 > URL: https://issues.apache.org/jira/browse/HBASE-19490 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Chia-Ping Tsai > Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7, 2.0.0-beta-1 > > Attachments: HBASE-19490.branch-1.v0.patch, HBASE-19490.v0.patch > > > Maybe we aren't waiting long enough for a slow executor? Or it is some kind > of race. Test usually passes. > [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.01 > s <<< FAILURE! - in org.apache.hadoop.hbase.quotas.TestRateLimiter > [ERROR] > testOverconsumptionFixedIntervalRefillStrategy(org.apache.hadoop.hbase.quotas.TestRateLimiter) > Time elapsed: 0.028 s <<< FAILURE! > java.lang.AssertionError: expected:<1000> but was:<999> > at > org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:122) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19471) Fix remaining Checkstyle errors in hbase-thrift
[ https://issues.apache.org/jira/browse/HBASE-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310658#comment-16310658 ] Chia-Ping Tsai commented on HBASE-19471: +1 > Fix remaining Checkstyle errors in hbase-thrift > --- > > Key: HBASE-19471 > URL: https://issues.apache.org/jira/browse/HBASE-19471 > Project: HBase > Issue Type: Sub-task > Components: Thrift >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Attachments: HBASE-19471.master.001.patch > > > Some Checkstyle errors are left in the *hbase-thrift* module and should be > fixed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19490) Rare failure in TestRateLimiter
[ https://issues.apache.org/jira/browse/HBASE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310656#comment-16310656 ] Hudson commented on HBASE-19490: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1052 (See [https://builds.apache.org/job/HBase-1.2-IT/1052/]) HBASE-19490 Rare failure in TestRateLimiter (chia7712: rev 19777252581c085e78177e1c147d0d79a77ab748) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java > Rare failure in TestRateLimiter > --- > > Key: HBASE-19490 > URL: https://issues.apache.org/jira/browse/HBASE-19490 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Chia-Ping Tsai > Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7, 2.0.0-beta-1 > > Attachments: HBASE-19490.branch-1.v0.patch, HBASE-19490.v0.patch > > > Maybe we aren't waiting long enough for a slow executor? Or it is some kind > of race. Test usually passes. > [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.01 > s <<< FAILURE! - in org.apache.hadoop.hbase.quotas.TestRateLimiter > [ERROR] > testOverconsumptionFixedIntervalRefillStrategy(org.apache.hadoop.hbase.quotas.TestRateLimiter) > Time elapsed: 0.028 s <<< FAILURE! > java.lang.AssertionError: expected:<1000> but was:<999> > at > org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:122) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310657#comment-16310657 ] Jingyun Tian commented on HBASE-19358: -- [~appy] attached:D > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split-logic-new.jpg, split-logic-old.jpg, split-table.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12904507/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * > hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is > *_hbase.regionserver.wal.max.splitters * the number of region the hlog > contains_*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279809#comment-16279809 ] Jingyun Tian edited comment on HBASE-19358 at 1/4/18 2:55 AM: -- [~carp84] here is my test result: Split one 512MB HLog on a single regionserver !https://issues.apache.org/jira/secure/attachment/12904505/split-1-log.png! we can see in most situation new logic has a better performance than the old one. The motivation I do this improvement is when a cluster has to restart, if there are too many regions per region, the restart is prone to failure and we have to split one hlog each time to avoid errors. So I test when restart the whole cluster, how many throughput it can reach with different thread count. Throughput when we restart a cluster, which has 18 regionservers and 18 datanodes !https://issues.apache.org/jira/secure/attachment/12904504/split_test_result.png! blue series represent the throughput of the cluster has 2 regions and regions per rs, while red series has 4 regions, regions per rs and orange series has 8 regions and per rs. This is the table if the chart is not clear: !https://issues.apache.org/jira/secure/attachment/12904508/split-table.png! Depend on this chart, I think the time cost when you restart the whole cluster is not related to the thread count. More regions this Hlog contains, more time it will cost to split. was (Author: tianjingyun): [~carp84] here is my test result: Split one 512MB HLog on a single regionserver !https://issues.apache.org/jira/secure/attachment/12904505/split-1-log.png! we can see in most situation new logic has a better performance than the old one. The motivation I do this improvement is when a cluster has to restart, if there are too many regions per region, the restart is prone to failure and we have to split one hlog each time to avoid errors. So I test when restart the whole cluster, how many throughput it can reach with different thread count. Throughput when we restart a cluster, which has 18 regionservers and 18 datanodes !https://issues.apache.org/jira/secure/attachment/12904504/split_test_result.png! blue series represent the throughput of the cluster has 2 regions and regions per rs, while red series has 4 regions, regions per rs and orange series has 8 regions and per rs. This is the table if the chart is not clear: !https://issues.apache.org/jira/secure/attachment/12903001/split-table.png! Depend on this chart, I think the time cost when you restart the whole cluster is not related to the thread count. More regions this Hlog contains, more time it will cost to split. > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split-logic-new.jpg, split-logic-old.jpg, split-table.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12904507/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * >
[jira] [Comment Edited] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279809#comment-16279809 ] Jingyun Tian edited comment on HBASE-19358 at 1/4/18 2:55 AM: -- [~carp84] here is my test result: Split one 512MB HLog on a single regionserver !https://issues.apache.org/jira/secure/attachment/12904505/split-1-log.png! we can see in most situation new logic has a better performance than the old one. The motivation I do this improvement is when a cluster has to restart, if there are too many regions per region, the restart is prone to failure and we have to split one hlog each time to avoid errors. So I test when restart the whole cluster, how many throughput it can reach with different thread count. Throughput when we restart a cluster, which has 18 regionservers and 18 datanodes !https://issues.apache.org/jira/secure/attachment/12904504/split_test_result.png! blue series represent the throughput of the cluster has 2 regions and regions per rs, while red series has 4 regions, regions per rs and orange series has 8 regions and per rs. This is the table if the chart is not clear: !https://issues.apache.org/jira/secure/attachment/12903001/split-table.png! Depend on this chart, I think the time cost when you restart the whole cluster is not related to the thread count. More regions this Hlog contains, more time it will cost to split. was (Author: tianjingyun): [~carp84] here is my test result: Split one 512MB HLog on a single regionserver !https://issues.apache.org/jira/secure/attachment/12904505/split-1-log.png! we can see in most situation new logic has a better performance than the old one. The motivation I do this improvement is when a cluster has to restart, if there are too many regions per region, the restart is prone to failure and we have to split one hlog each time to avoid errors. So I test when restart the whole cluster, how many throughput it can reach with different thread count. Throughput when we restart a cluster, which has 18 regionservers and 18 datanodes !https://issues.apache.org/jira/secure/attachment/12902999/split_test_result.png! blue series represent the throughput of the cluster has 2 regions and regions per rs, while red series has 4 regions, regions per rs and orange series has 8 regions and per rs. This is the table if the chart is not clear: !https://issues.apache.org/jira/secure/attachment/12903001/split-table.png! Depend on this chart, I think the time cost when you restart the whole cluster is not related to the thread count. More regions this Hlog contains, more time it will cost to split. > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split-logic-new.jpg, split-logic-old.jpg, split-table.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12904507/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * >
[jira] [Comment Edited] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279809#comment-16279809 ] Jingyun Tian edited comment on HBASE-19358 at 1/4/18 2:54 AM: -- [~carp84] here is my test result: Split one 512MB HLog on a single regionserver !https://issues.apache.org/jira/secure/attachment/12904505/split-1-log.png! we can see in most situation new logic has a better performance than the old one. The motivation I do this improvement is when a cluster has to restart, if there are too many regions per region, the restart is prone to failure and we have to split one hlog each time to avoid errors. So I test when restart the whole cluster, how many throughput it can reach with different thread count. Throughput when we restart a cluster, which has 18 regionservers and 18 datanodes !https://issues.apache.org/jira/secure/attachment/12902999/split_test_result.png! blue series represent the throughput of the cluster has 2 regions and regions per rs, while red series has 4 regions, regions per rs and orange series has 8 regions and per rs. This is the table if the chart is not clear: !https://issues.apache.org/jira/secure/attachment/12903001/split-table.png! Depend on this chart, I think the time cost when you restart the whole cluster is not related to the thread count. More regions this Hlog contains, more time it will cost to split. was (Author: tianjingyun): [~carp84] here is my test result: Split one 512MB HLog on a single regionserver !https://issues.apache.org/jira/secure/attachment/12902996/split-1-log.png! we can see in most situation new logic has a better performance than the old one. The motivation I do this improvement is when a cluster has to restart, if there are too many regions per region, the restart is prone to failure and we have to split one hlog each time to avoid errors. So I test when restart the whole cluster, how many throughput it can reach with different thread count. Throughput when we restart a cluster, which has 18 regionservers and 18 datanodes !https://issues.apache.org/jira/secure/attachment/12902999/split_test_result.png! blue series represent the throughput of the cluster has 2 regions and regions per rs, while red series has 4 regions, regions per rs and orange series has 8 regions and per rs. This is the table if the chart is not clear: !https://issues.apache.org/jira/secure/attachment/12903001/split-table.png! Depend on this chart, I think the time cost when you restart the whole cluster is not related to the thread count. More regions this Hlog contains, more time it will cost to split. > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split-logic-new.jpg, split-logic-old.jpg, split-table.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12904507/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * >
[jira] [Updated] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-19358: - Description: The way we splitting log now is like the following figure: !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! The problem is the OutputSink will write the recovered edits during splitting log, which means it will create one WriterAndPath for each region and retain it until the end. If the cluster is small and the number of regions per rs is large, it will create too many HDFS streams at the same time. Then it is prone to failure since each datanode need to handle too many streams. Thus I come up with a new way to split log. !https://issues.apache.org/jira/secure/attachment/12904507/split-logic-new.jpg! We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, we will pick the largest EntryBuffer and write it to a file (close the writer after finish). Then after we read all entries into memory, we will start a writeAndCloseThreadPool, it starts a certain number of threads to write all buffers to files. Thus it will not create HDFS streams more than *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. The biggest benefit is we can control the number of streams we create during splitting log, it will not exceeds *_hbase.regionserver.wal.max.splitters * hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is *_hbase.regionserver.wal.max.splitters * the number of region the hlog contains_*. was: The way we splitting log now is like the following figure: !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! The problem is the OutputSink will write the recovered edits during splitting log, which means it will create one WriterAndPath for each region and retain it until the end. If the cluster is small and the number of regions per rs is large, it will create too many HDFS streams at the same time. Then it is prone to failure since each datanode need to handle too many streams. Thus I come up with a new way to split log. !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg! We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, we will pick the largest EntryBuffer and write it to a file (close the writer after finish). Then after we read all entries into memory, we will start a writeAndCloseThreadPool, it starts a certain number of threads to write all buffers to files. Thus it will not create HDFS streams more than *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. The biggest benefit is we can control the number of streams we create during splitting log, it will not exceeds *_hbase.regionserver.wal.max.splitters * hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is *_hbase.regionserver.wal.max.splitters * the number of region the hlog contains_*. > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split-logic-new.jpg, split-logic-old.jpg, split-table.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12904507/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams
[jira] [Updated] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-19358: - Description: The way we splitting log now is like the following figure: !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! The problem is the OutputSink will write the recovered edits during splitting log, which means it will create one WriterAndPath for each region and retain it until the end. If the cluster is small and the number of regions per rs is large, it will create too many HDFS streams at the same time. Then it is prone to failure since each datanode need to handle too many streams. Thus I come up with a new way to split log. !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg! We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, we will pick the largest EntryBuffer and write it to a file (close the writer after finish). Then after we read all entries into memory, we will start a writeAndCloseThreadPool, it starts a certain number of threads to write all buffers to files. Thus it will not create HDFS streams more than *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. The biggest benefit is we can control the number of streams we create during splitting log, it will not exceeds *_hbase.regionserver.wal.max.splitters * hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is *_hbase.regionserver.wal.max.splitters * the number of region the hlog contains_*. was: The way we splitting log now is like the following figure: !https://issues.apache.org/jira/secure/attachment/12902997/split-logic-old.jpg! The problem is the OutputSink will write the recovered edits during splitting log, which means it will create one WriterAndPath for each region and retain it until the end. If the cluster is small and the number of regions per rs is large, it will create too many HDFS streams at the same time. Then it is prone to failure since each datanode need to handle too many streams. Thus I come up with a new way to split log. !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg! We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, we will pick the largest EntryBuffer and write it to a file (close the writer after finish). Then after we read all entries into memory, we will start a writeAndCloseThreadPool, it starts a certain number of threads to write all buffers to files. Thus it will not create HDFS streams more than *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. The biggest benefit is we can control the number of streams we create during splitting log, it will not exceeds *_hbase.regionserver.wal.max.splitters * hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is *_hbase.regionserver.wal.max.splitters * the number of region the hlog contains_*. > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split-logic-new.jpg, split-logic-old.jpg, split-table.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12904506/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams
[jira] [Updated] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-19358: - Attachment: split-table.png split-logic-new.jpg split-logic-old.jpg > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split-logic-new.jpg, split-logic-old.jpg, split-table.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12902997/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * > hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is > *_hbase.regionserver.wal.max.splitters * the number of region the hlog > contains_*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19490) Rare failure in TestRateLimiter
[ https://issues.apache.org/jira/browse/HBASE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19490: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review. [~tedyu] > Rare failure in TestRateLimiter > --- > > Key: HBASE-19490 > URL: https://issues.apache.org/jira/browse/HBASE-19490 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Chia-Ping Tsai > Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7, 2.0.0-beta-1 > > Attachments: HBASE-19490.branch-1.v0.patch, HBASE-19490.v0.patch > > > Maybe we aren't waiting long enough for a slow executor? Or it is some kind > of race. Test usually passes. > [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.01 > s <<< FAILURE! - in org.apache.hadoop.hbase.quotas.TestRateLimiter > [ERROR] > testOverconsumptionFixedIntervalRefillStrategy(org.apache.hadoop.hbase.quotas.TestRateLimiter) > Time elapsed: 0.028 s <<< FAILURE! > java.lang.AssertionError: expected:<1000> but was:<999> > at > org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:122) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-19358: - Attachment: split-1-log.png split_test_result.png > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch, split-1-log.png, > split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12902997/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * > hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is > *_hbase.regionserver.wal.max.splitters * the number of region the hlog > contains_*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-11020) TestFromClientSide3#testHTableExistsMethodSingleRegionMultipleGets missing Test annotation
[ https://issues.apache.org/jira/browse/HBASE-11020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-11020: --- Resolution: Duplicate Status: Resolved (was: Patch Available) see HBASE-19195 > TestFromClientSide3#testHTableExistsMethodSingleRegionMultipleGets missing > Test annotation > -- > > Key: HBASE-11020 > URL: https://issues.apache.org/jira/browse/HBASE-11020 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 0.96.2, 0.98.1, 0.99.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Attachments: HBASE-11020.00.patch > > > Per title. I see nothing obvious in the commit log indicating the omission is > by design. Test passes when annotation added. Maybe this was one of those > pesky flakey tests? I'm guessing not... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19613) Review of WALSplitter Class
[ https://issues.apache.org/jira/browse/HBASE-19613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19613: - Resolution: Fixed Status: Resolved (was: Patch Available) > Review of WALSplitter Class > --- > > Key: HBASE-19613 > URL: https://issues.apache.org/jira/browse/HBASE-19613 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19613.1.patch, HBASE-19613.2.patch > > > * Use ArrayList instead LinkedList > * Use Apache Commons where appropriate > * Parameterize and improve logging -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19613) Review of WALSplitter Class
[ https://issues.apache.org/jira/browse/HBASE-19613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19613: - Fix Version/s: 2.0.0-beta-2 > Review of WALSplitter Class > --- > > Key: HBASE-19613 > URL: https://issues.apache.org/jira/browse/HBASE-19613 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19613.1.patch, HBASE-19613.2.patch > > > * Use ArrayList instead LinkedList > * Use Apache Commons where appropriate > * Parameterize and improve logging -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19701) Close without justification following succesful open
stack created HBASE-19701:
-
Summary: Close without justification following succesful open
Key: HBASE-19701
URL: https://issues.apache.org/jira/browse/HBASE-19701
Project: HBase
Issue Type: Bug
Reporter: stack
Priority: Critical
Fix For: 2.0.0-beta-2
[~jmspaggi] conjured an interesting condition where we close a region soon
after open WITHOUT seemingly saying why (It looks like Master is asking for
region CLOSE but that is not clear looking at RegionServer log).
Here is log snippet from https://pastebin.com/0r76Y6ap (in case the pastebin
evaporates)
{code}
2017-12-31 09:54:20,864 INFO
[PostOpenDeployTasks:f49f3cbb7f3db4cf96c7eb3b0cf83869]
regionserver.HRegionServer: Post open deploy tasks for
TestTable,0408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
2017-12-31 09:54:20,870 INFO [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
regionserver.CompactingMemStore: Setting in-memory flush size threshold to
13421772 and immutable segments index to be of type CHUNK_MAP
2017-12-31 09:54:20,870 INFO [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
regionserver.HStore: Memstore class name is
org.apache.hadoop.hbase.regionserver.CompactingMemStore
2017-12-31 09:54:20,870 INFO [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
hfile.CacheConfig: Created cacheConfig for info:
blockCache=LruBlockCache{blockCount=0, currentSize=2454760,
freeSize=3347745560, maxSize=3350200320, heapSize=2454760, minSize=3182690304,
minFactor=0.95, multiSize=1591345152, multiFactor=0.5, singleSize=795672576,
singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false,
cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2017-12-31 09:54:20,872 INFO [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00;
throttle point 2684354560; major period 60480, major jitter 0,50, min
locality to compact 0,00; tiered compaction: max_age 9223372036854775807,
incoming window min 6, compaction policy for tiered window
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
single output for minor true, compaction window factory
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2017-12-31 09:54:20,903 INFO [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
regionserver.CompactingMemStore: Setting in-memory flush size threshold to
13421772 and immutable segments index to be of type CHUNK_MAP
2017-12-31 09:54:20,904 INFO [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
regionserver.HStore: Memstore class name is
org.apache.hadoop.hbase.regionserver.CompactingMemStore
2017-12-31 09:54:20,904 INFO [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
hfile.CacheConfig: Created cacheConfig for info:
blockCache=LruBlockCache{blockCount=0, currentSize=2454760,
freeSize=3347745560, maxSize=3350200320, heapSize=2454760, minSize=3182690304,
minFactor=0.95, multiSize=1591345152, multiFactor=0.5, singleSize=795672576,
singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false,
cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2017-12-31 09:54:20,905 INFO [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00;
throttle point 2684354560; major period 60480, major jitter 0,50, min
locality to compact 0,00; tiered compaction: max_age 9223372036854775807,
incoming window min 6, compaction policy for tiered window
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
single output for minor true, compaction window factory
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2017-12-31 09:54:20,929 INFO [RS_OPEN_REGION-node1:16020-1]
regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
region=TestTable,0262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
2017-12-31 09:54:20,956 INFO [RS_OPEN_REGION-node1:16020-0]
regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
region=TestTable,0188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
2017-12-31 09:54:20,991 INFO [RS_OPEN_REGION-node1:16020-1]
regionserver.HRegion: Onlined 166b9c45d7724f72fd126adb4445d6ec; next
sequenceid=22861
2017-12-31 09:54:20,998 INFO
[PostOpenDeployTasks:166b9c45d7724f72fd126adb4445d6ec]
regionserver.HRegionServer: Post open deploy tasks for
TestTable,0262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
[jira] [Commented] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310646#comment-16310646 ] Appy commented on HBASE-19358: -- Reattach the images please :) > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-18619-branch-2-v2.patch, HBASE-18619-branch-2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-v1.patch, HBASE-19358-v4.patch, > HBASE-19358-v5.patch, HBASE-19358-v6.patch, HBASE-19358-v7.patch, > HBASE-19358-v8.patch, HBASE-19358.patch > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12902997/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12902998/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * > hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is > *_hbase.regionserver.wal.max.splitters * the number of region the hlog > contains_*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[
https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310636#comment-16310636
]
Appy commented on HBASE-15809:
--
Btw, you can try the sample UI by unzipping it and running local python server
({{python -m SimpleHTTPServer}}).
> Basic Replication WebUI
> ---
>
> Key: HBASE-15809
> URL: https://issues.apache.org/jira/browse/HBASE-15809
> Project: HBase
> Issue Type: New Feature
> Components: Replication, UI
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Appy
>Priority: Minor
> Fix For: 2.0.0, 1.5.0
>
> Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png,
> HBASE-15809-v1.patch, rep_web_ui.zip
>
>
> At the moment the only way to have some insight on replication from the webui
> is looking at zkdump and metrics.
> the basic information useful to get started debugging are: peer information
> and the view of WALs offsets for each peer.
> https://reviews.apache.org/r/47275/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-15809: - Attachment: rep_web_ui.zip Attached rep_web_ui.zip containing frontend code (html and js). The UI is specifically designed for scale i.e. ~500-1000 nodes cluster. At that scale, it's impossible to put numbers/raw data in top level UI. It's rather colorful :) Files description: *.csv : Fake data used for designing the Ui. Assuming cluster size of 500nodes. *.min.js: Standard js libs index.html: The code i wrote to generate UI from fake data. This is the main part. Contains js too. What's missing is backend to actually collect the data and feed into UI. We had few general ideas, but none seemed trivial. Let me put the ones i remember here: # UI uses ajax to query RegionServers' jmx every X sec. X depends on cluster size and will likely be in order of minutes for 1000 node cluster. Might lead to interesting challenges with browser. Idk. # Each RS sends selective replication metrics to master, and UI fetches everything in one sweep. Since data can be ~MBs for large cluster, can put non-negligible load on master. Refresh rate of UI should be set accrodingly. > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.5.0 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch, rep_web_ui.zip > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19650) ExpiredMobFileCleaner has wrong logic about TTL check
[ https://issues.apache.org/jira/browse/HBASE-19650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenxu updated HBASE-19650: --- Description: when MobUtils.cleanExpiredMobFiles execute, fileDate will be less than expireDate even if they were on the same day, because expireDate contains MILLISECOND info, so ExpiredMobFileCleaner will clean more files than it should. (was: If today is 2017-12-28 00:00:01, and TTL is set to 86400, when MobUtils.cleanExpiredMobFiles execute, expireDate will be 1514304000749, but fileDate is 151430400. So the fileDate before the expireDate and mobfiles generated in is 2017-12-27 will all are deleted. But in fact, we want to delete files before 2017-12-27.) > ExpiredMobFileCleaner has wrong logic about TTL check > - > > Key: HBASE-19650 > URL: https://issues.apache.org/jira/browse/HBASE-19650 > Project: HBase > Issue Type: Bug > Components: mob >Reporter: chenxu >Assignee: chenxu > Attachments: HBASE-19650-master-v1.patch, HBASE-19650-master-v2.patch > > > when MobUtils.cleanExpiredMobFiles execute, fileDate will be less than > expireDate even if they were on the same day, because expireDate contains > MILLISECOND info, so ExpiredMobFileCleaner will clean more files than it > should. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19699) Revisit class ReplicationSourceManager
[ https://issues.apache.org/jira/browse/HBASE-19699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310628#comment-16310628 ] Zheng Hu commented on HBASE-19699: -- OK, can do it later. Let's focus on merging HBASE-19397. > Revisit class ReplicationSourceManager > -- > > Key: HBASE-19699 > URL: https://issues.apache.org/jira/browse/HBASE-19699 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Affects Versions: HBASE-19397 >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: HBASE-19397 > > > Code cleanup and simplification for ReplicationSourceManager. > Will start this work after HBASE-19636 resolved. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18806:
-
Hadoop Flags: Reviewed
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch,
> HBASE-18806.v3.patch, HBASE-18806.v3.patch, HBASE-18806.v4.patch,
> HBASE-18806.v4.patch, HBASE-18806.v5.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310620#comment-16310620
]
Zheng Hu commented on HBASE-18806:
--
Pushed to master & branch-2. Thanks [~tedyu] for reviewing.
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch,
> HBASE-18806.v3.patch, HBASE-18806.v3.patch, HBASE-18806.v4.patch,
> HBASE-18806.v4.patch, HBASE-18806.v5.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Release Note:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now we can guarantee that once the procedure is done, the peer modification
has already taken effect on all RSes.
We have also abstracted a storage layer for replication peer/queue manangement,
and refactored the upper layer to remove zk related naming/code/comment.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18806:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch,
> HBASE-18806.v3.patch, HBASE-18806.v3.patch, HBASE-18806.v4.patch,
> HBASE-18806.v4.patch, HBASE-18806.v5.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18806:
-
Fix Version/s: 2.0.0-beta-2
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch,
> HBASE-18806.v3.patch, HBASE-18806.v3.patch, HBASE-18806.v4.patch,
> HBASE-18806.v4.patch, HBASE-18806.v5.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19613) Review of WALSplitter Class
[ https://issues.apache.org/jira/browse/HBASE-19613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310611#comment-16310611 ] Appy commented on HBASE-19613: -- I thought the changed removed stack trace all together. Didn't notice that you made it part of log.warn. Sgtm. Pushing to master and branch-2. Thanks for the patch beluga. > Review of WALSplitter Class > --- > > Key: HBASE-19613 > URL: https://issues.apache.org/jira/browse/HBASE-19613 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-19613.1.patch, HBASE-19613.2.patch > > > * Use ArrayList instead LinkedList > * Use Apache Commons where appropriate > * Parameterize and improve logging -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-18806:
-
Affects Version/s: (was: 2.0.0-alpha-2)
2.0.0-beta-2
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-beta-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch,
> HBASE-18806.v3.patch, HBASE-18806.v3.patch, HBASE-18806.v4.patch,
> HBASE-18806.v4.patch, HBASE-18806.v5.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18625) Splitting of region with replica, doesn't update region list in serverHolding. A server crash leads to overlap.
[
https://issues.apache.org/jira/browse/HBASE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310584#comment-16310584
]
huaxiang sun commented on HBASE-18625:
--
Thanks [~apurtell].
> Splitting of region with replica, doesn't update region list in
> serverHolding. A server crash leads to overlap.
> ---
>
> Key: HBASE-18625
> URL: https://issues.apache.org/jira/browse/HBASE-18625
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.6
>Reporter: Igloo
>Assignee: huaxiang sun
> Fix For: 1.4.1, 1.5.0
>
> Attachments: HBASE-18625-branch-1-v001.patch,
> HBASE-18625-branch-1-v002.patch
>
>
> The situation can appear in following steps in release hbase1.2.6
> 1. create 'testtable', 'info', {REGION_REPLICATION=>2}
> 2. write somerecords into 'testtable'
> 3. split the table 'testtable'
> 4. after the spliting, the serverHoldings in RegionStates still holds the
> regioninfo for the replica of parent region
> 5. restart the regionserver where the parent replica-region located
> 6. the offlined replica of parent region will be assigned in
> ServerCrashProcedure.
> hbase hbck 'testtable‘
> ERROR: Region { meta => null, hdfs => null, deployed =>
> qabb-qa-hdp-hbase1,16020,1503022958093;testtable,,1503022907686_0001.42d11cfe195b3cc4d08b2c078a687f6d
> ., replicaId => 1 } not in META, but deployed on
> qabb-qa-hdp-hbase1,16020,1503022958093
> 18 ERROR: No regioninfo in Meta or HDFS. { meta => null, hdfs => null,
> deployed =>
> qabb-qa-hdp-hbase1,16020,1503022958093;testtable,,1503022907686_0001.42d11cfe
>195b3cc4d08b2c078a687f6d., replicaId => 1 }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19697) Remove TestReplicationAdminUsingProcedure
[ https://issues.apache.org/jira/browse/HBASE-19697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19697: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HBASE-19397 Status: Resolved (was: Patch Available) Pushed to branch HBASE-19397. Thanks [~openinx] for reviewing. > Remove TestReplicationAdminUsingProcedure > - > > Key: HBASE-19697 > URL: https://issues.apache.org/jira/browse/HBASE-19697 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: HBASE-19397 > > Attachments: HBASE-19697-HBASE-19397.patch > > > It is useless now since we have already on procedure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310555#comment-16310555
]
huaxiang sun commented on HBASE-17319:
--
I checked the master branch, it seems that the issue is there. After that, the
new test failed for the master. Let me post the patch for the master.
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17319-branch-1.patch, 17319.stack,
> HBASE-17319-branch-1.patch, HBASE-17319-branch-1.patch, HBASE-17319.patch,
> TestTruncateTableProcedure-output.tar.gz
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19490) Rare failure in TestRateLimiter
[ https://issues.apache.org/jira/browse/HBASE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310552#comment-16310552 ] Ted Yu commented on HBASE-19490: lgtm > Rare failure in TestRateLimiter > --- > > Key: HBASE-19490 > URL: https://issues.apache.org/jira/browse/HBASE-19490 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Chia-Ping Tsai > Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7, 2.0.0-beta-1 > > Attachments: HBASE-19490.branch-1.v0.patch, HBASE-19490.v0.patch > > > Maybe we aren't waiting long enough for a slow executor? Or it is some kind > of race. Test usually passes. > [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.01 > s <<< FAILURE! - in org.apache.hadoop.hbase.quotas.TestRateLimiter > [ERROR] > testOverconsumptionFixedIntervalRefillStrategy(org.apache.hadoop.hbase.quotas.TestRateLimiter) > Time elapsed: 0.028 s <<< FAILURE! > java.lang.AssertionError: expected:<1000> but was:<999> > at > org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:122) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19490) Rare failure in TestRateLimiter
[ https://issues.apache.org/jira/browse/HBASE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310541#comment-16310541 ] Chia-Ping Tsai commented on HBASE-19490: Ping for reviews. > Rare failure in TestRateLimiter > --- > > Key: HBASE-19490 > URL: https://issues.apache.org/jira/browse/HBASE-19490 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Chia-Ping Tsai > Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7, 2.0.0-beta-1 > > Attachments: HBASE-19490.branch-1.v0.patch, HBASE-19490.v0.patch > > > Maybe we aren't waiting long enough for a slow executor? Or it is some kind > of race. Test usually passes. > [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.01 > s <<< FAILURE! - in org.apache.hadoop.hbase.quotas.TestRateLimiter > [ERROR] > testOverconsumptionFixedIntervalRefillStrategy(org.apache.hadoop.hbase.quotas.TestRateLimiter) > Time elapsed: 0.028 s <<< FAILURE! > java.lang.AssertionError: expected:<1000> but was:<999> > at > org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:122) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19691) Do not require ADMIN permission for obtaining ClusterStatus
[ https://issues.apache.org/jira/browse/HBASE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-19691: --- Resolution: Fixed Hadoop Flags: Reviewed Release Note: This change reverts an unintentional requirement for global ADMIN permission to obtain cluster status from the active HMaster. Status: Resolved (was: Patch Available) > Do not require ADMIN permission for obtaining ClusterStatus > --- > > Key: HBASE-19691 > URL: https://issues.apache.org/jira/browse/HBASE-19691 > Project: HBase > Issue Type: Bug >Reporter: Romil Choksi >Assignee: Josh Elser >Priority: Critical > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: HBASE-19691.001.branch-2.patch > > > Appears to be a regression introduced by HBASE-19131. Operations that attempt > to obtain the `status` from the HMaster now fail if the requesting user > doesn't have global ADMIN permission. > Discussion: > https://lists.apache.org/thread.html/f1cd2a50e5c460879c97043790b33aa375cd6b217455d611c3417e3d@%3Cdev.hbase.apache.org%3E > Thanks to Romil for letting us know about this one. > FYI [~stack] [~chia7712]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19691) Do not require ADMIN permission for obtaining ClusterStatus
[ https://issues.apache.org/jira/browse/HBASE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-19691: --- Fix Version/s: 1.5.0 > Do not require ADMIN permission for obtaining ClusterStatus > --- > > Key: HBASE-19691 > URL: https://issues.apache.org/jira/browse/HBASE-19691 > Project: HBase > Issue Type: Bug >Reporter: Romil Choksi >Assignee: Josh Elser >Priority: Critical > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: HBASE-19691.001.branch-2.patch > > > Appears to be a regression introduced by HBASE-19131. Operations that attempt > to obtain the `status` from the HMaster now fail if the requesting user > doesn't have global ADMIN permission. > Discussion: > https://lists.apache.org/thread.html/f1cd2a50e5c460879c97043790b33aa375cd6b217455d611c3417e3d@%3Cdev.hbase.apache.org%3E > Thanks to Romil for letting us know about this one. > FYI [~stack] [~chia7712]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19691) Do not require ADMIN permission for obtaining ClusterStatus
[ https://issues.apache.org/jira/browse/HBASE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310487#comment-16310487 ] Josh Elser commented on HBASE-19691: TestMemstoreLABWithoutPool failed because the test OOME'd. Going to push this. Thanks for the review, Chia-Ping! > Do not require ADMIN permission for obtaining ClusterStatus > --- > > Key: HBASE-19691 > URL: https://issues.apache.org/jira/browse/HBASE-19691 > Project: HBase > Issue Type: Bug >Reporter: Romil Choksi >Assignee: Josh Elser >Priority: Critical > Fix For: 1.4.1, 2.0.0-beta-2 > > Attachments: HBASE-19691.001.branch-2.patch > > > Appears to be a regression introduced by HBASE-19131. Operations that attempt > to obtain the `status` from the HMaster now fail if the requesting user > doesn't have global ADMIN permission. > Discussion: > https://lists.apache.org/thread.html/f1cd2a50e5c460879c97043790b33aa375cd6b217455d611c3417e3d@%3Cdev.hbase.apache.org%3E > Thanks to Romil for letting us know about this one. > FYI [~stack] [~chia7712]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19691) Do not require ADMIN permission for obtaining ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310473#comment-16310473
]
Hadoop QA commented on HBASE-19691:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
1s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
7s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
53s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 11s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 42s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMemstoreLABWithoutPool |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19691 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904475/HBASE-19691.001.branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 37cf2041982e 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 97dc7d87cd |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/10876/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
[jira] [Updated] (HBASE-19596) RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes
[ https://issues.apache.org/jira/browse/HBASE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19596: --- Attachment: HBASE-19596.v3.patch rebase again..Too sisyphean for me. > RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes > - > > Key: HBASE-19596 > URL: https://issues.apache.org/jira/browse/HBASE-19596 > Project: HBase > Issue Type: Task > Components: Client >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19596.v0.patch, HBASE-19596.v1.patch, > HBASE-19596.v2.patch, HBASE-19596.v3.patch > > > ServerLoad/RegionLoad/ClusterLoad are deprecated now. This issue will > deprecate all related methods in our Public classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19700) assertion failure with client 1.4, cluster 1.2.3 and table with presplit
Clément Guillaume created HBASE-19700:
-
Summary: assertion failure with client 1.4, cluster 1.2.3 and
table with presplit
Key: HBASE-19700
URL: https://issues.apache.org/jira/browse/HBASE-19700
Project: HBase
Issue Type: Bug
Affects Versions: 1.4.0
Reporter: Clément Guillaume
A system assertion (which is active by default when running
maven-failsafe-plugin) is failing when a 1.4 client is talking to a 1.2.3
cluster for table with preslits. I believe the [1.4 client is meant to be
compatible with a 1.2
cluster|http://mail-archives.apache.org/mod_mbox/hbase-dev/201711.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E]
{code}
@Test
public void test() throws IOException{
Configuration hbaseConfig = HBaseConfiguration.create();
hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
Connection connection = ConnectionFactory.createConnection(hbaseConfig);
TableName tableName = TableName.valueOf("AssertionTest");
Admin admin = connection.getAdmin();
if(!admin.tableExists(tableName)){
HTableDescriptor htable = new HTableDescriptor(tableName);
htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
admin.createTable(htable, splitPoints);
System.out.println("table created");
}
Table table = connection.getTable(tableName);
ResultScanner scanner = table.getScanner(new Scan());
scanner.iterator().hasNext(); // Exception thrown here
}
{code}
{code}
java.lang.RuntimeException: java.lang.AssertionError
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:227)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:277)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:438)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
at
org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:92)
at [...]
Caused by: java.lang.AssertionError
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:484)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1324)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
... 30 more
{code}
[Email
thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201712.mbox/%3CCALte62z-0xxhQiefeRc_3xs-bhj1VZU%2BBtd47m-KfPZb02Tpcw%40mail.gmail.com%3E]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19634) Add permission check for executeProcedures in AccessController
[ https://issues.apache.org/jira/browse/HBASE-19634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310414#comment-16310414 ] Duo Zhang commented on HBASE-19634: --- Ping [~stack]. I'm going to commit this if no objections. Maybe we should add a notice in our ref guide to tell users that, for a secure HBase cluster, the user who start HMaster should be configured as super user if you use different user to start HMaster and HRegionserver. > Add permission check for executeProcedures in AccessController > -- > > Key: HBASE-19634 > URL: https://issues.apache.org/jira/browse/HBASE-19634 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19634-HBASE-19397-v1.patch, > HBASE-19634-HBASE-19397-v1.patch, HBASE-19634-HBASE-19397-v2.patch, > HBASE-19634-HBASE-19397.patch > > > This is important, the actual refresh on RS is trigger by the > executeProcedure call and it will pass some information. These information > should not be fully trusted since anyone can all this method. We need to make > sure that the actual data/state for a replication peer is always loaded from > the replication storage, not from the parameter of the executeProcedure call. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19684) BlockCacheKey toString Performance
[
https://issues.apache.org/jira/browse/HBASE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310410#comment-16310410
]
Mike Drob commented on HBASE-19684:
---
Yea, looking at the bytecode we can confirm that concat operator uses a
StringBuilder under the hood. It compiles to essentially the same code, except
without an initial buffer size like you noted.
> BlockCacheKey toString Performance
> --
>
> Key: HBASE-19684
> URL: https://issues.apache.org/jira/browse/HBASE-19684
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19684.1.patch, HBASE-19684.2.patch
>
>
> {code:titile=BlockCacheKey.java}
> @Override
> public String toString() {
> return String.format("%s_%d", hfileName, offset);
> }
> {code}
> I found through bench-marking that the following code is 10x faster.
> {code:titi\le=BlockCacheKey.java}
> @Override
> public String toString() {
> return hfileName.concat("_").concat(Long.toString(offset));
> }
> {code}
> Normally it wouldn't matter for a _toString()_ method, but this is comes into
> play because {{MemcachedBlockCache}} uses it.
> {code:title=MemcachedBlockCache.java}
> @Override
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf) {
> if (buf instanceof HFileBlock) {
> client.add(cacheKey.toString(), MAX_SIZE, (HFileBlock) buf, tc);
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("MemcachedBlockCache can not cache Cacheable's of type "
> + buf.getClass().toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
