[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035956#comment-15035956
]
Heng Chen commented on HBASE-14895:
---
It seems we could add some code in StoreScanner.updateReaders, right?
{code: title=StoreScanner.updateReaders}
this.heap = null;
+StoreFile recentFlush = getRecentFlushStoreFile();
+if (recentFlush != null) {
+ seek(recentFlush.getFirstKey());
+}
{code}
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035921#comment-15035921
]
Heng Chen commented on HBASE-14905:
---
You are right, in distributed cluster, tasks could not share static variable.
patch looks good. +1
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: 2.0.0, 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Fix For: 2.0.0
>
> Attachments: HBASE-14905.patch, test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Relax region read lock for compactions
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035939#comment-15035939 ] Hudson commented on HBASE-14575: FAILURE: Integrated in HBase-1.3-IT #350 (See [https://builds.apache.org/job/HBase-1.3-IT/350/]) HBASE-14575 Relax region read lock for compactions (Nick and Ted) (tedyu: rev 3b6ae17a72059e8fad9487ab5c02e47d04378e66) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Relax region read lock for compactions > -- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 14575-test-case.patch, 14575-v1.patch, 14575-v2.patch, > 14575-v3.patch, 14575-v4.patch, 14575-v5.patch, 14575-v6.txt, 14575-v6.txt, > 14575-v6.txt, 14575-v7.txt, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. > Here is summary from parent issue: > Another idea is we can reduce the scope of when the read lock is held during > compaction. In theory the compactor only needs a region read lock while > deciding what files to compact and at the time of committing the compaction. > We're protected from the case of region close events because compactions are > checking (every 10k bytes written) if the store has been closed in order to > abort in such a case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14906) Improvements on FlushLargeStoresPolicy
[
https://issues.apache.org/jira/browse/HBASE-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036035#comment-15036035
]
Yu Li commented on HBASE-14906:
---
Thanks for the quick response [~Apache9]!
About config in table description, I think we need to preserve a way for
experts to override the default value, in case they really need to set a
*smaller* value than {{region.getMemstoreFlushSize() / familyNumber}} for their
use case (setting the global config to a smaller value won't take effect by
current design). Agree?
> Improvements on FlushLargeStoresPolicy
> --
>
> Key: HBASE-14906
> URL: https://issues.apache.org/jira/browse/HBASE-14906
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14906.patch
>
>
> When checking FlushLargeStoragePolicy, found below possible improving points:
> 1. Currently in selectStoresToFlush, we will do the selection no matter how
> many actual families, which is not necessary for one single family
> 2. Default value for hbase.hregion.percolumnfamilyflush.size.lower.bound
> could not fit in all cases, and requires user to know details of the
> implementation to properly set it. We propose to use
> "hbase.hregion.memstore.flush.size/column_family_number" instead:
> {noformat}
>
> hbase.hregion.percolumnfamilyflush.size.lower.bound
> 16777216
>
> If FlushLargeStoresPolicy is used and there are multiple column families,
> then every time that we hit the total memstore limit, we find out all the
> column families whose memstores exceed a "lower bound" and only flush them
> while retaining the others in memory. The "lower bound" will be
> "hbase.hregion.memstore.flush.size / column_family_number" by default
> unless value of this property is larger than that. If none of the families
> have their memstore size more than lower bound, all the memstores will be
> flushed (just as usual).
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14819:
--
Attachment: 14819.branch-1.txt
What I committed to branch-1 and branch-1.2.
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819.branch-1.txt, 14819.total.master.changes.patch, 14819v2.txt, Screen
> Shot 2015-11-16 at 11.37.41 PM.png, itag.txt, more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
> org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.setupRegistry(ConnectionManager.java:886)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:692)
> at
>
[jira] [Updated] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14909:
--
Attachment: 14909.branch-1.txt
14909.txt
What I pushed to branch-1.2+
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-14909.
---
Resolution: Fixed
Assignee: stack
Fix Version/s: 1.3.0
1.2.0
2.0.0
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14908) TestRowCounter flakey especially on branch-1
[ https://issues.apache.org/jira/browse/HBASE-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036313#comment-15036313 ] stack commented on HBASE-14908: --- Test came in here: HBASE-12447 > TestRowCounter flakey especially on branch-1 > > > Key: HBASE-14908 > URL: https://issues.apache.org/jira/browse/HBASE-14908 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack > Attachments: 14908.patch > > > It fails a bunch, about one in five runs: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ > Looking at log it seems to be making good progress without issue but the test > is running four mapreduce tasks and it wants to finish inside three minutes > (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14908) TestRowCounter flakey especially on branch-1
[ https://issues.apache.org/jira/browse/HBASE-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036312#comment-15036312 ] stack commented on HBASE-14908: --- Pushed patch to 1.2+. Will leave issue open a while to see if it helps. > TestRowCounter flakey especially on branch-1 > > > Key: HBASE-14908 > URL: https://issues.apache.org/jira/browse/HBASE-14908 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack > Attachments: 14908.patch > > > It fails a bunch, about one in five runs: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ > Looking at log it seems to be making good progress without issue but the test > is running four mapreduce tasks and it wants to finish inside three minutes > (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036091#comment-15036091 ] ramkrishna.s.vasudevan commented on HBASE-14895: Yes something like that only. We need to clearly identify the latest files and just seek to that. With HBASE-13082 in place we just need to do for flushes only. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036179#comment-15036179 ] Vladimir Rodionov commented on HBASE-14030: --- On a hbase:backup table. We do need this table to restore backups, only to do backups. It is up to user to decide when and how to backup this table. > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v3.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14908) TestRowCounter flakey especially on branch-1
[ https://issues.apache.org/jira/browse/HBASE-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14908: -- Attachment: 14908.patch Let me just try change the category on the test from medium to large so has more time to run and see if that stabilizes it. > TestRowCounter flakey especially on branch-1 > > > Key: HBASE-14908 > URL: https://issues.apache.org/jira/browse/HBASE-14908 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack > Attachments: 14908.patch > > > It fails a bunch, about one in five runs: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ > Looking at log it seems to be making good progress without issue but the test > is running four mapreduce tasks and it wants to finish inside three minutes > (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Relax region read lock for compactions
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036324#comment-15036324 ] Hudson commented on HBASE-14575: SUCCESS: Integrated in HBase-1.3 #410 (See [https://builds.apache.org/job/HBase-1.3/410/]) HBASE-14575 Relax region read lock for compactions (Nick and Ted) (tedyu: rev 3b6ae17a72059e8fad9487ab5c02e47d04378e66) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Relax region read lock for compactions > -- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 14575-test-case.patch, 14575-v1.patch, 14575-v2.patch, > 14575-v3.patch, 14575-v4.patch, 14575-v5.patch, 14575-v6.txt, 14575-v6.txt, > 14575-v6.txt, 14575-v7.txt, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. > Here is summary from parent issue: > Another idea is we can reduce the scope of when the read lock is held during > compaction. In theory the compactor only needs a region read lock while > deciding what files to compact and at the time of committing the compaction. > We're protected from the case of region close events because compactions are > checking (every 10k bytes written) if the store has been closed in order to > abort in such a case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14749) Make changes to region_mover.rb to use RegionMover Java tool
[ https://issues.apache.org/jira/browse/HBASE-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036135#comment-15036135 ] Samir Ahmic commented on HBASE-14749: - Tested on 5 node cluster moving few hundred regions with 'rolling-restart.sh --graceful', also i have tested graceful_stop.sh and region_mover.rb with different options. No major issues found, logs look fine. I did not test scripts in any disruptive scenarios . Overall +1 in my opinion. > Make changes to region_mover.rb to use RegionMover Java tool > > > Key: HBASE-14749 > URL: https://issues.apache.org/jira/browse/HBASE-14749 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 2.0.0 > > Attachments: HBASE-14749-v2.patch, HBASE-14749-v3.patch, > HBASE-14749-v3.patch, HBASE-14749-v4.patch, HBASE-14749-v5.patch, > HBASE-14749.patch, HBASE-14749.patch > > > With HBASE-13014 in, we can now replace the ruby script such that it invokes > the Java Tool. Also expose timeout and no-ack mode which were added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036262#comment-15036262
]
stack commented on HBASE-14819:
---
So, let me make a patch for branch-1. Here is all I did for the master branch
on this issue trying to get it to pass:
{code}
3534e1f HBASE-14819 hbase-it tests failing with OOME; permgen ; ADDENDUM...
more heap and permgen for failsafe runs
7979ac4 HBASE-14819 hbase-it tests failing with OOME; permgen
4a60c25 HBASE-14819 hbase-it tests failing with OOME; permgen -- DEBUGGING
97ffb19 HBASE-14819 hbase-it tests failing with OOME; permgen; DEBUG
26423e4 Revert "HBASE-14819 hbase-it tests failing with OOME; permgen ADDENDUM"
ea48ef8 HBASE-14819 hbase-it tests failing with OOME; permgen ADDENDUM
7dc4f12 HBASE-14819 hbase-it tests failing with OOME: permgen
{code}
Let me make up a coherent patch to backport.
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819v2.txt, Screen Shot 2015-11-16 at 11.37.41 PM.png, itag.txt,
> more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036161#comment-15036161 ] Vladimir Rodionov commented on HBASE-14030: --- Updated review board with the latest patch. > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v3.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14909) NPE testing for RIT
stack created HBASE-14909:
-
Summary: NPE testing for RIT
Key: HBASE-14909
URL: https://issues.apache.org/jira/browse/HBASE-14909
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: stack
Master build just failed because TestMasterFailover failed with NPE testing for
RIT:
{code}
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
{code}
Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036159#comment-15036159
]
Hadoop QA commented on HBASE-14905:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775272/HBASE-14905.patch
against master branch at commit aa41232a877d7a8485bc361fd62150d7c094e9a4.
ATTACHMENT ID: 12775272
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16731//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16731//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16731//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16731//console
This message is automatically generated.
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: 2.0.0, 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Fix For: 2.0.0
>
> Attachments: HBASE-14905.patch, test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because
[jira] [Updated] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14819:
--
Attachment: 14819.total.master.changes.patch
File of all the changes I tried on master in order.
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819.total.master.changes.patch, 14819v2.txt, Screen Shot 2015-11-16 at
> 11.37.41 PM.png, itag.txt, more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
> org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.setupRegistry(ConnectionManager.java:886)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:692)
> at
>
[jira] [Updated] (HBASE-14575) Relax region read lock for compactions
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14575: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 2.0.0 Status: Resolved (was: Patch Available) Unit test suite results are quite clean. Thanks for the reviews. > Relax region read lock for compactions > -- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0, 1.3.0 > > Attachments: 14575-test-case.patch, 14575-v1.patch, 14575-v2.patch, > 14575-v3.patch, 14575-v4.patch, 14575-v5.patch, 14575-v6.txt, 14575-v6.txt, > 14575-v6.txt, 14575-v7.txt, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. > Here is summary from parent issue: > Another idea is we can reduce the scope of when the read lock is held during > compaction. In theory the compactor only needs a region read lock while > deciding what files to compact and at the time of committing the compaction. > We're protected from the case of region close events because compactions are > checking (every 10k bytes written) if the store has been closed in order to > abort in such a case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14749) Make changes to region_mover.rb to use RegionMover Java tool
[ https://issues.apache.org/jira/browse/HBASE-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036175#comment-15036175 ] Abhishek Singh Chouhan commented on HBASE-14749: Thanks @asamir :) > Make changes to region_mover.rb to use RegionMover Java tool > > > Key: HBASE-14749 > URL: https://issues.apache.org/jira/browse/HBASE-14749 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 2.0.0 > > Attachments: HBASE-14749-v2.patch, HBASE-14749-v3.patch, > HBASE-14749-v3.patch, HBASE-14749-v4.patch, HBASE-14749-v5.patch, > HBASE-14749.patch, HBASE-14749.patch > > > With HBASE-13014 in, we can now replace the ruby script such that it invokes > the Java Tool. Also expose timeout and no-ack mode which were added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036213#comment-15036213
]
Ted Yu commented on HBASE-14905:
nit:
The log message below should be consistent with the other messages - 'number of
versions set to'
{code}
284 LOG.info("Version set to " + versions);
{code}
Can the regression test be part of the patch ?
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: 2.0.0, 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Fix For: 2.0.0
>
> Attachments: HBASE-14905.patch, test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14899) Create custom Streaming ReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-14899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036328#comment-15036328
]
Enis Soztutar commented on HBASE-14899:
---
Is this for replication v1 or v2? The interface between the
{{ReplicationEndpoint}} and {{ReplicationSource}} is not suitable for streaming
in v1 I think.
We can still do streaming-like thing within ReplicationEndpoint.replicate(),
although there will be a barrier at the end of the method call.
> Create custom Streaming ReplicationEndpoint
> ---
>
> Key: HBASE-14899
> URL: https://issues.apache.org/jira/browse/HBASE-14899
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Cody Marcel
>Assignee: Cody Marcel
> Fix For: 2.0.0
>
>
> Create a custom Replication Endpoint that streams WALEntry directly through
> client API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14819:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
We are passing now mostly and no more OOME. Let me apply a patch to other
branches also.
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819v2.txt, Screen Shot 2015-11-16 at 11.37.41 PM.png, itag.txt,
> more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
> org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.setupRegistry(ConnectionManager.java:886)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:692)
> at
>
[jira] [Created] (HBASE-14908) TestRowCounter flakey especially on branch-1
stack created HBASE-14908: - Summary: TestRowCounter flakey especially on branch-1 Key: HBASE-14908 URL: https://issues.apache.org/jira/browse/HBASE-14908 Project: HBase Issue Type: Sub-task Components: flakey, test Affects Versions: 1.2.0 Reporter: stack Assignee: stack It fails a bunch, about one in five runs: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ Looking at log it seems to be making good progress without issue but the test is running four mapreduce tasks and it wants to finish inside three minutes (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14541) TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many splits and few retries
[
https://issues.apache.org/jira/browse/HBASE-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036285#comment-15036285
]
Dima Spivak commented on HBASE-14541:
-
Can this one be closed out, [~stack]?
> TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many
> splits and few retries
> --
>
> Key: HBASE-14541
> URL: https://issues.apache.org/jira/browse/HBASE-14541
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Matteo Bertozzi
> Attachments: HBASE-14541-test.patch, HBASE-14541-v0.patch,
> HBASE-14541-v0.patch
>
>
> This one seems worth a dig. We seem to be making progress but here is what we
> are trying to load which seems weird:
> {code}
> 2015-10-01 17:19:41,322 INFO [main] mapreduce.LoadIncrementalHFiles(360):
> Split occured while grouping HFiles, retry attempt 10 with 4 files remaining
> to group or split
> 2015-10-01 17:19:41,323 ERROR [main] mapreduce.LoadIncrementalHFiles(402):
> -
> Bulk load aborted with some files not yet loaded:
> -
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.top
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.top
> {code}
> Whats that about?
> Making note here. Will keep an eye on this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036319#comment-15036319
]
stack commented on HBASE-14819:
---
Backport seems warranted though itbll usually passes on branch-1*. This
branch-1.3 it fail was the JVM struggling it seems. See end of here:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3-IT/it.test=IntegrationTestBigLinkedList,jdk=latest1.7,label=Hadoop/350/artifact/hbase-it/target/failsafe-reports/org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList-output.txt
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819.branch-1.txt, 14819.total.master.changes.patch, 14819v2.txt, Screen
> Shot 2015-11-16 at 11.37.41 PM.png, itag.txt, more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
>
[jira] [Updated] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14904:
--
Attachment: hbase-14904_v2.patch
Attaching rebased v2.
[~apurtell] what do you think about this one? The binary compat issue only
affects Phoenix (only known WALCellCodec implementation outside HBase). We can
re-break the binary compat between 0.98.16 and 0.98.17. We did not do any 1.x
release yet with HBASE-14501.
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14899) Create custom Streaming ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-14899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036399#comment-15036399 ] Cody Marcel commented on HBASE-14899: - I am still working on the details. AFAIK, V2 (HBASE-14379) is just a set of guidelines that we want to follow while improving replication. Has there been some propositions for interface changes? Please let me know if there are, but I am not aware of any. This patch would definitely conform to the requirements outlined in the V2 jira. It would be plugable and as minimally invasive as possible. You are correct though. It will probably be closer to "streaming-like" rather than actual streaming. I think I'd still work with the WALEdit. > Create custom Streaming ReplicationEndpoint > --- > > Key: HBASE-14899 > URL: https://issues.apache.org/jira/browse/HBASE-14899 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.0.0 >Reporter: Cody Marcel >Assignee: Cody Marcel > Fix For: 2.0.0 > > > Create a custom Replication Endpoint that streams WALEntry directly through > client API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14899) Create custom Streaming ReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-14899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036412#comment-15036412
]
Enis Soztutar commented on HBASE-14899:
---
Yes, I was assuming the RE interface be changed to allow an iterator based
model where WAL reading, streaming replication and ACK'ing back the WAL offset
points are concurrent allowing pipelining. Current {{RE.replicate(batch of
edits)}} API should see some changes I imagine.
But if should be fine to still have a v1 RE endpoint to do the streaming writes
as much as possible.
> Create custom Streaming ReplicationEndpoint
> ---
>
> Key: HBASE-14899
> URL: https://issues.apache.org/jira/browse/HBASE-14899
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Cody Marcel
>Assignee: Cody Marcel
> Fix For: 2.0.0
>
>
> Create a custom Replication Endpoint that streams WALEntry directly through
> client API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14908) TestRowCounter flakey especially on branch-1
[ https://issues.apache.org/jira/browse/HBASE-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036474#comment-15036474 ] Hudson commented on HBASE-14908: FAILURE: Integrated in HBase-1.2-IT #322 (See [https://builds.apache.org/job/HBase-1.2-IT/322/]) HBASE-14908 TestRowCounter flakey especially on branch-1 (stack: rev 1de1909c137d48f81b7c86a27897c78d14fe138a) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java > TestRowCounter flakey especially on branch-1 > > > Key: HBASE-14908 > URL: https://issues.apache.org/jira/browse/HBASE-14908 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack > Attachments: 14908.patch > > > It fails a bunch, about one in five runs: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ > Looking at log it seems to be making good progress without issue but the test > is running four mapreduce tasks and it wants to finish inside three minutes > (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036473#comment-15036473
]
Hudson commented on HBASE-14909:
FAILURE: Integrated in HBase-1.2-IT #322 (See
[https://builds.apache.org/job/HBase-1.2-IT/322/])
HBASE-14909 NPE testing for RIT (stack: rev
53379191c14fe491d213454b7e9a3ba1c332a278)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14908) TestRowCounter flakey especially on branch-1
[ https://issues.apache.org/jira/browse/HBASE-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036397#comment-15036397 ] Hudson commented on HBASE-14908: FAILURE: Integrated in HBase-1.3-IT #351 (See [https://builds.apache.org/job/HBase-1.3-IT/351/]) HBASE-14908 TestRowCounter flakey especially on branch-1 (stack: rev 81de832732ffabf30add0251fb2690470cb41524) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java > TestRowCounter flakey especially on branch-1 > > > Key: HBASE-14908 > URL: https://issues.apache.org/jira/browse/HBASE-14908 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack > Attachments: 14908.patch > > > It fails a bunch, about one in five runs: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ > Looking at log it seems to be making good progress without issue but the test > is running four mapreduce tasks and it wants to finish inside three minutes > (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036398#comment-15036398
]
Hudson commented on HBASE-14819:
FAILURE: Integrated in HBase-1.3-IT #351 (See
[https://builds.apache.org/job/HBase-1.3-IT/351/])
HBASE-14819 hbase-it tests failing with OOME; permgen (stack: rev
f730a375e3c67b8e3bbfa2ec4e4488f35bd2db0d)
* hbase-it/pom.xml
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819.branch-1.txt, 14819.total.master.changes.patch, 14819v2.txt, Screen
> Shot 2015-11-16 at 11.37.41 PM.png, itag.txt, more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
> org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43)
> at
>
[jira] [Created] (HBASE-14910) Create base hbase-mapred and hbase-io modules, move HTableInputFormat and HTableOutputFormat
Nate Edel created HBASE-14910: - Summary: Create base hbase-mapred and hbase-io modules, move HTableInputFormat and HTableOutputFormat Key: HBASE-14910 URL: https://issues.apache.org/jira/browse/HBASE-14910 Project: HBase Issue Type: Sub-task Reporter: Nate Edel Assignee: Nate Edel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14912) Move functionality needed to read WAL without RegionServer dependency
Nate Edel created HBASE-14912: - Summary: Move functionality needed to read WAL without RegionServer dependency Key: HBASE-14912 URL: https://issues.apache.org/jira/browse/HBASE-14912 Project: HBase Issue Type: Sub-task Reporter: Nate Edel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14913) Enable testing I/O without regionserver or threads.
Nate Edel created HBASE-14913: - Summary: Enable testing I/O without regionserver or threads. Key: HBASE-14913 URL: https://issues.apache.org/jira/browse/HBASE-14913 Project: HBase Issue Type: Sub-task Reporter: Nate Edel Assignee: Nate Edel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14914) Move functionality needed to handle Snapshots without RegionServer dependency
Nate Edel created HBASE-14914: - Summary: Move functionality needed to handle Snapshots without RegionServer dependency Key: HBASE-14914 URL: https://issues.apache.org/jira/browse/HBASE-14914 Project: HBase Issue Type: Sub-task Reporter: Nate Edel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14911) Move functionality needed to write HFiles without RegionServer dependency
Nate Edel created HBASE-14911: - Summary: Move functionality needed to write HFiles without RegionServer dependency Key: HBASE-14911 URL: https://issues.apache.org/jira/browse/HBASE-14911 Project: HBase Issue Type: Sub-task Reporter: Nate Edel Assignee: Nate Edel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14873) Problems around BoundedByteBufferPool providing direct buffers
[
https://issues.apache.org/jira/browse/HBASE-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036612#comment-15036612
]
stack commented on HBASE-14873:
---
bq. To tell the truth I didn't expect to be agreed with the V2 path.
Why sir? What is wrong with the approach especially if use BBUtil where we go
to Unsafe if applicable.
bq. BTW, once again, the Oracle implementation does a little more things than
ByteBufferUtils.copyFromBufferToArray.
You have a point. What you reckon [~anoop.hbase]?
bq. . In other words there are situations where we should intentionally use the
right buffer in the right place.
Makes sense. I think the places where BBBP have been deployed so far make
sense. The latest instance, using them pulling request off the socket, is still
TODO.
This issue that you filed is a dirty bug we need to fix regardless, right?
Thanks [~ikeda]
> Problems around BoundedByteBufferPool providing direct buffers
> --
>
> Key: HBASE-14873
> URL: https://issues.apache.org/jira/browse/HBASE-14873
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: HBASE-14873-V2.patch, HBASE-14873.patch,
> HBASE-14873.patch, HBASE-14873.patch
>
>
> HBASE-13819 made BoundedByteBufferPool provide direct buffers.
> See RpcServer.java:
> {code}
> ...
> class Call implements RpcCallContext {
> protected synchronized void setResponse(...) {
> ...
> this.cellBlock = ipcUtil.buildCellBlock(..., reservoir);
> ...
> bc = new BufferChain(..., this.cellBlock);
> if (connection.useWrap) {
> bc = wrapWithSasl(bc);
> }
> ...
> private BufferChain wrapWithSasl(BufferChain bc) throws IOException {
> ...
> byte[] responseBytes = bc.getBytes();
> ...
> {code}
> {{cellBlock}} is expected to be a direct buffer retrieved from {{reservoir}}
> (but not always), and {{bc}} may be composed of both direct and non-direct
> buffers.
> And then, see BufferChain.java:
> {code}
> byte [] getBytes() {
> ...
> for (ByteBuffer bb: this.buffers) {
> System.arraycopy(bb.array(), ...);
> {code}
> A direct buffer doesn't give its array, and will throw
> UnsupportedOperationException.
> Another problem; {{cellBlock}} is allowed to be a non-direct buffer, and
> after use it will be put to {{reservoir}}, mixing direct and non-direct
> buffers in the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036349#comment-15036349
]
stack commented on HBASE-14909:
---
Here is the build that failed:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/lastCompletedBuild/jdk=latest1.7,label=Hadoop/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testMetaInTransitionWhenMasterFailover/
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14541) TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many splits and few retries
[
https://issues.apache.org/jira/browse/HBASE-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14541:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
2.0.0
Status: Resolved (was: Patch Available)
Resolving. We don't see this any more (thanks [~dimaspivak]) Let me make sure
it is in 1.2 too.
> TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many
> splits and few retries
> --
>
> Key: HBASE-14541
> URL: https://issues.apache.org/jira/browse/HBASE-14541
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14541-test.patch, HBASE-14541-v0.patch,
> HBASE-14541-v0.patch
>
>
> This one seems worth a dig. We seem to be making progress but here is what we
> are trying to load which seems weird:
> {code}
> 2015-10-01 17:19:41,322 INFO [main] mapreduce.LoadIncrementalHFiles(360):
> Split occured while grouping HFiles, retry attempt 10 with 4 files remaining
> to group or split
> 2015-10-01 17:19:41,323 ERROR [main] mapreduce.LoadIncrementalHFiles(402):
> -
> Bulk load aborted with some files not yet loaded:
> -
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.top
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.top
> {code}
> Whats that about?
> Making note here. Will keep an eye on this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed Increments, CheckAndPuts, batch operations
[ https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036353#comment-15036353 ] Bryan Beaudreault commented on HBASE-14460: --- Thank you [~stack]. One piece of feedback we have so far on CDAP after only a couple days using it is that it is installed on a column family and thus requires each CF it is installed on to have unlimited versions. We understand why it requires unlimited versions in its current state to roll up the increments, but this is awful when you have a column which is incremented in a CF with other non-increment data. All of that other data will have unlimited versions as well. If something like CDAP were incorporated into hbase native I imagine there are better ways to do this roll up. I just wanted to make sure I voiced this as a great thing to improve upon: in the hbase-native version, we shouldn't need to configure an entire CF to have unlimited versions if just 1 or 2 columns are being incremented. Also, the vast majority of our increments are write-only (we don't care about the result at increment time). But a small handful do care about that value. So having two different versions (one read-write, one write-only) would be great. > [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed > Increments, CheckAndPuts, batch operations > --- > > Key: HBASE-14460 > URL: https://issues.apache.org/jira/browse/HBASE-14460 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 14460.txt, region_lock.png > > > As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation > between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of > sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to > 'catch up' to our current point before we can read the last Increment value > that we need to update. > We can say that our Increment is just done wrong, we should just be writing > Increments and summing on read, but checkAndPut as well as batching > operations have the same issue. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11843) MapReduce classes shouldn't be in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-11843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036441#comment-15036441 ] Nate Edel commented on HBASE-11843: --- This is looking bigger, and probably difficult to review especially coming from a relative newb. I've broken things up into subtasks, and this should be easier to review. > MapReduce classes shouldn't be in hbase-server > -- > > Key: HBASE-11843 > URL: https://issues.apache.org/jira/browse/HBASE-11843 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Keegan Witt >Assignee: Nate Edel >Priority: Critical > Fix For: 2.0.0 > > > I'm opening this to continue the discussion I started a while back: > http://mail-archives.apache.org/mod_mbox/hbase-user/201405.mbox/%3ccamuu0w_xooxeg779rrfjturau+uxeavunzxkw9dxfo-gh5y...@mail.gmail.com%3E. > To summarize, I think the MapReduce classes used by clients (like > TableMapper, TableReducer, etc) don't belong in hbase-server. This forces > the user to pull in a rather large artifact for a relatively small number of > classes. These should either be put in hbase-client, or possibly an artifact > of their own (like the hbase-mapreduce idea Enis suggested). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036475#comment-15036475
]
Hudson commented on HBASE-14819:
FAILURE: Integrated in HBase-1.2-IT #322 (See
[https://builds.apache.org/job/HBase-1.2-IT/322/])
HBASE-14819 hbase-it tests failing with OOME; permgen (stack: rev
32fc0e788fc4957149c0bbc77b62f851147fa8ce)
* hbase-it/pom.xml
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819.branch-1.txt, 14819.total.master.changes.patch, 14819v2.txt, Screen
> Shot 2015-11-16 at 11.37.41 PM.png, itag.txt, more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
> org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43)
> at
>
[jira] [Commented] (HBASE-14541) TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many splits and few retries
[
https://issues.apache.org/jira/browse/HBASE-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036507#comment-15036507
]
stack commented on HBASE-14541:
---
I pushed this to 1.2 too.
> TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many
> splits and few retries
> --
>
> Key: HBASE-14541
> URL: https://issues.apache.org/jira/browse/HBASE-14541
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14541-test.patch, HBASE-14541-v0.patch,
> HBASE-14541-v0.patch
>
>
> This one seems worth a dig. We seem to be making progress but here is what we
> are trying to load which seems weird:
> {code}
> 2015-10-01 17:19:41,322 INFO [main] mapreduce.LoadIncrementalHFiles(360):
> Split occured while grouping HFiles, retry attempt 10 with 4 files remaining
> to group or split
> 2015-10-01 17:19:41,323 ERROR [main] mapreduce.LoadIncrementalHFiles(402):
> -
> Bulk load aborted with some files not yet loaded:
> -
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.top
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.top
> {code}
> Whats that about?
> Making note here. Will keep an eye on this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14541) TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many splits and few retries
[
https://issues.apache.org/jira/browse/HBASE-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14541:
--
Fix Version/s: 1.2.0
> TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many
> splits and few retries
> --
>
> Key: HBASE-14541
> URL: https://issues.apache.org/jira/browse/HBASE-14541
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14541-test.patch, HBASE-14541-v0.patch,
> HBASE-14541-v0.patch
>
>
> This one seems worth a dig. We seem to be making progress but here is what we
> are trying to load which seems weird:
> {code}
> 2015-10-01 17:19:41,322 INFO [main] mapreduce.LoadIncrementalHFiles(360):
> Split occured while grouping HFiles, retry attempt 10 with 4 files remaining
> to group or split
> 2015-10-01 17:19:41,323 ERROR [main] mapreduce.LoadIncrementalHFiles(402):
> -
> Bulk load aborted with some files not yet loaded:
> -
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.top
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.top
> {code}
> Whats that about?
> Making note here. Will keep an eye on this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14866) VerifyReplication should use peer configuration in peer connection
[
https://issues.apache.org/jira/browse/HBASE-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036536#comment-15036536
]
Gary Helmling commented on HBASE-14866:
---
[~chenheng], [~eclark]: any thoughts on the latest patch?
> VerifyReplication should use peer configuration in peer connection
> --
>
> Key: HBASE-14866
> URL: https://issues.apache.org/jira/browse/HBASE-14866
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14866.patch, HBASE-14866_v1.patch,
> hbase-14866-v4.patch, hbase-14866_v2.patch, hbase-14866_v3.patch
>
>
> VerifyReplication uses the replication peer's configuration to construct the
> ZooKeeper quorum address for the peer connection. However, other
> configuration properties in the peer's configuration are dropped. It should
> merge all configuration properties from the {{ReplicationPeerConfig}} when
> creating the peer connection and obtaining a credentials for the peer cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Attachment: HBASE-14901.3.patch > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Status: Open (was: Patch Available) > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Status: Patch Available (was: Open) Fixed checkstyle errors (bad imports.) Fixed some existing checkstyle errors (bad import order, no private constructor on EncryptionUtil and MobUtils). Fixed "protected" LOG on EncryptionUtil to private > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14902: -- Attachment: 14902.patch Suggested patch. What you think [~appy]? Do we need a writeup somewhere on the recent checkstyle changes? For instance, I did not know about the diff created by a patch now is dumped in console output (any chance of adding line numbers to this dump or is that too much). And would be good to find a note that rather than '@throw IOException', we suggest just remove the javadoc since it adds nothing. > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Fix For: 2.0.0 > > Attachments: 14902.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035473#comment-15035473
]
Heng Chen commented on HBASE-14905:
---
Hi, [~vishk]
I can't reproduce it locally, if row in source has different cells with row in
target, it will throw exception, So GoodRows counter is always 0 and badRows
count is 1.
{code: title=Result.java}
768 if (res1.size() != res2.size()) {
769 throw new Exception("This row doesn't have the same number of KVs: "
770 + res1.toString() + " compared to " + res2.toString());
771}
{code}
Wound you mind upload your test patch?
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Reporter: Vishal Khandelwal
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035520#comment-15035520
]
Vishal Khandelwal commented on HBASE-14905:
---
Sure. [~chenheng]. I will update the test patch.
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal reassigned HBASE-14905:
-
Assignee: Vishal Khandelwal
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14907) NPE of MobUtils.hasMobColumns in Build failed in Jenkins: HBase-Trunk_matrix » latest1.8,Hadoop #513
Jingcheng Du created HBASE-14907: Summary: NPE of MobUtils.hasMobColumns in Build failed in Jenkins: HBase-Trunk_matrix » latest1.8,Hadoop #513 Key: HBASE-14907 URL: https://issues.apache.org/jira/browse/HBASE-14907 Project: HBase Issue Type: Bug Components: mob Reporter: Jingcheng Du Assignee: Jingcheng Du NPE is thrown when rollback the failures of table creation. 1. Table is being created, get issues when creating fs layout. 2. Rollback this creation and trying to delete the data from fs. It tries to delete the mob dir and needs to ask HMaster about the HTableDescriptor, and at that time the table dir had been deleted and no HTableDescriptor can be found. In this patch, it directly checks if the mob directory is existing instead of checking the HTableDescriptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14905:
--
Attachment: test.patch
upload my test patch.
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Reporter: Vishal Khandelwal
> Attachments: test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14832) Ensure write paths work with ByteBufferedCells in case of compaction
[ https://issues.apache.org/jira/browse/HBASE-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035482#comment-15035482 ] Hudson commented on HBASE-14832: FAILURE: Integrated in HBase-Trunk_matrix #523 (See [https://builds.apache.org/job/HBase-Trunk_matrix/523/]) HBASE-14832 Ensure write paths work with ByteBufferedCells in case of (ramkrishna: rev aa41232a877d7a8485bc361fd62150d7c094e9a4) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/test/RedundantKVGenerator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java > Ensure write paths work with ByteBufferedCells in case of compaction > > > Key: HBASE-14832 > URL: https://issues.apache.org/jira/browse/HBASE-14832 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14832.patch, HBASE-14832_1.patch > > > Currently any cell coming out of offheap Bucketcache while compaction does a > copy using the getXXXArray() API since write path does not work with BBCells. > This JIRA is aimed at changing the write path to support BBCells so that this > copy is avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14869) Better request latency histograms
[
https://issues.apache.org/jira/browse/HBASE-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035506#comment-15035506
]
Hadoop QA commented on HBASE-14869:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774947/14869-v1-2.0.txt
against master branch at commit aa41232a877d7a8485bc361fd62150d7c094e9a4.
ATTACHMENT ID: 12774947
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+metaSplitTimeHisto =
metricsRegistry.newTimeHistogram(META_SPLIT_TIME_NAME, META_SPLIT_TIME_DESC);
+metaSplitSizeHisto =
metricsRegistry.newSizeHistogram(META_SPLIT_SIZE_NAME, META_SPLIT_SIZE_DESC);
+.addCounter(Interns.info(name + "_" + getRangeType() + "_" + prior
+ "-" + getRange()[i], desc), val);
+ private final long[] ranges =
{1,3,10,30,100,300,1000,3000,1,3,6,12,30,60};
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16728//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16728//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16728//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16728//console
This message is automatically generated.
> Better request latency histograms
> -
>
> Key: HBASE-14869
> URL: https://issues.apache.org/jira/browse/HBASE-14869
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 14869-test-0.98.txt, 14869-v1-0.98.txt,
> 14869-v1-2.0.txt, 14869-v2-0.98.txt, 14869-v3-0.98.txt, 14869-v4-0.98.txt,
> AppendSizeTime.png, Get.png
>
>
> I just discussed this with a colleague.
> The get, put, etc, histograms that each region server keeps are somewhat
> useless (depending on what you want to achieve of course), as they are
> aggregated and calculated by each region server.
> It would be better to record the number of requests in certainly latency
> bands in addition to what we do now.
> For example the number of gets that took 0-5ms, 6-10ms, 10-20ms, 20-50ms,
> 50-100ms, 100-1000ms, > 1000ms, etc. (just as an example, should be
> configurable).
> That way we can do further calculations after the fact, and answer questions
> like: How often did we miss our SLA? Percentage of requests that missed an
> SLA, etc.
> Comments?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-14865:
-
Attachment: HBASE-14865-master-v7.patch
Since trying re-submitting to trigger test-patch didn't work 2 times,
resubmitting v6 as v7 (rebased).
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-master-v2.patch,
> HBASE-14865-master-v3.patch, HBASE-14865-master-v4.patch,
> HBASE-14865-master-v5.patch, HBASE-14865-master-v6.patch,
> HBASE-14865-master-v7.patch, HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036664#comment-15036664
]
Hudson commented on HBASE-14909:
FAILURE: Integrated in HBase-1.2-IT #323 (See
[https://builds.apache.org/job/HBase-1.2-IT/323/])
HBASE-14909 NPE testing for RIT; addendum -- forgot import (stack: rev
f2ce05d8a134c5d4c12f1dafa787b0e416460939)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14915) Hanging test : org.apache.hadoop.hbase.mapreduce.TestImportExport
stack created HBASE-14915: - Summary: Hanging test : org.apache.hadoop.hbase.mapreduce.TestImportExport Key: HBASE-14915 URL: https://issues.apache.org/jira/browse/HBASE-14915 Project: HBase Issue Type: Sub-task Components: hangingTests Reporter: stack This test hangs a bunch: Here is latest: https://builds.apache.org/job/HBase-1.2/418/jdk=latest1.7,label=Hadoop/consoleText -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036688#comment-15036688
]
Hudson commented on HBASE-14909:
SUCCESS: Integrated in HBase-1.3-IT #352 (See
[https://builds.apache.org/job/HBase-1.3-IT/352/])
HBASE-14909 NPE testing for RIT (stack: rev
96e5f040369b3250750f44c8791535c62f494f6c)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
HBASE-14909 NPE testing for RIT; addendum -- forgot import (stack: rev
7f2e09f9c98ebc2b6b738efb13172b0c2c924f6f)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14866) VerifyReplication should use peer configuration in peer connection
[
https://issues.apache.org/jira/browse/HBASE-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036784#comment-15036784
]
Elliott Clark commented on HBASE-14866:
---
bq. transformClusterKey
Should transformClusterKey just use standardizeZKQuorumServerString rather than
having two different cases?
bq.TestHBaseConfiguration
Seems like the entry for shouldbemissing will have two prefixes. Is that
intended?
Most of these changes to HBaseConfiguration seem to be very replication
specific. Should we have a different class for replication based configuration,
so that HBaseConfiguration doesn't get too unwieldy?
> VerifyReplication should use peer configuration in peer connection
> --
>
> Key: HBASE-14866
> URL: https://issues.apache.org/jira/browse/HBASE-14866
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14866.patch, HBASE-14866_v1.patch,
> hbase-14866-v4.patch, hbase-14866_v2.patch, hbase-14866_v3.patch
>
>
> VerifyReplication uses the replication peer's configuration to construct the
> ZooKeeper quorum address for the peer connection. However, other
> configuration properties in the peer's configuration are dropped. It should
> merge all configuration properties from the {{ReplicationPeerConfig}} when
> creating the peer connection and obtaining a credentials for the peer cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036684#comment-15036684 ] Appy commented on HBASE-14902: -- +1 on patch. Yeah, explicitly telling everyone about changes in checkstyles and reporting will be really good. Does sending a mail to dev@ sound good? I can do that. Or anything else you have in mind. > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Fix For: 2.0.0 > > Attachments: 14902.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036720#comment-15036720 ] stack commented on HBASE-14902: --- I think mail to dev by the man who did the work would be good starting point. After any questions have all been resolved, we should add a little doc to refguide. Nice. > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Fix For: 2.0.0 > > Attachments: 14902.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036762#comment-15036762
]
Hadoop QA commented on HBASE-14904:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775356/hbase-14904_v2.patch
against master branch at commit da0cc598feab995eed12527d90805dd627674035.
ATTACHMENT ID: 12775356
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16732//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16732//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16732//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16732//console
This message is automatically generated.
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (HBASE-14541) TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many splits and few retries
[
https://issues.apache.org/jira/browse/HBASE-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036665#comment-15036665
]
Hudson commented on HBASE-14541:
FAILURE: Integrated in HBase-1.2-IT #323 (See
[https://builds.apache.org/job/HBase-1.2-IT/323/])
HBASE-14541 TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed (stack:
rev 2444047d076f64fbd8acf7684426101f5b1c0527)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
> TestHFileOutputFormat.testMRIncrementalLoadWithSplit failed due to too many
> splits and few retries
> --
>
> Key: HBASE-14541
> URL: https://issues.apache.org/jira/browse/HBASE-14541
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14541-test.patch, HBASE-14541-v0.patch,
> HBASE-14541-v0.patch
>
>
> This one seems worth a dig. We seem to be making progress but here is what we
> are trying to load which seems weird:
> {code}
> 2015-10-01 17:19:41,322 INFO [main] mapreduce.LoadIncrementalHFiles(360):
> Split occured while grouping HFiles, retry attempt 10 with 4 files remaining
> to group or split
> 2015-10-01 17:19:41,323 ERROR [main] mapreduce.LoadIncrementalHFiles(402):
> -
> Bulk load aborted with some files not yet loaded:
> -
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/ce11cbe2490d444d8958264004286aff.top
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.bottom
>
> hdfs://localhost:39540/user/jenkins/test-data/720ae36a-2495-456b-ba68-19e260685a35/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/30c58eeb23a6464da21117e6e1bc565c.top
> {code}
> Whats that about?
> Making note here. Will keep an eye on this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14719) Add metric for number of MasterProcWALs
[ https://issues.apache.org/jira/browse/HBASE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036789#comment-15036789 ] Elliott Clark commented on HBASE-14719: --- lgtm. If we add too many procedure based metrics we might want to add a wrapper for that only. However I don't think it's needed for just a couple of metrics yet. > Add metric for number of MasterProcWALs > --- > > Key: HBASE-14719 > URL: https://issues.apache.org/jira/browse/HBASE-14719 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Vrishal Kulkarni > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14719.1.patch, HBASE-14719.2.patch, > HBASE-14719.3.patch, HBASE-14719.patch > > > Lets add monitoring to this so that we can see when it starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14908) TestRowCounter flakey especially on branch-1
[ https://issues.apache.org/jira/browse/HBASE-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036813#comment-15036813 ] Hudson commented on HBASE-14908: FAILURE: Integrated in HBase-Trunk_matrix #524 (See [https://builds.apache.org/job/HBase-Trunk_matrix/524/]) HBASE-14908 TestRowCounter flakey especially on branch-1 (stack: rev 08f90f30b385bbf314f7fb014b810d86330d60b1) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java > TestRowCounter flakey especially on branch-1 > > > Key: HBASE-14908 > URL: https://issues.apache.org/jira/browse/HBASE-14908 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack > Attachments: 14908.patch > > > It fails a bunch, about one in five runs: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ > Looking at log it seems to be making good progress without issue but the test > is running four mapreduce tasks and it wants to finish inside three minutes > (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036812#comment-15036812
]
Hudson commented on HBASE-14909:
FAILURE: Integrated in HBase-Trunk_matrix #524 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/524/])
HBASE-14909 NPE testing for RIT (stack: rev
da0cc598feab995eed12527d90805dd627674035)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-14865:
-
Attachment: HBASE-14865-branch-1.patch
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14909) NPE testing for RIT
[
https://issues.apache.org/jira/browse/HBASE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036934#comment-15036934
]
Hudson commented on HBASE-14909:
SUCCESS: Integrated in HBase-1.3 #411 (See
[https://builds.apache.org/job/HBase-1.3/411/])
HBASE-14909 NPE testing for RIT (stack: rev
96e5f040369b3250750f44c8791535c62f494f6c)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
HBASE-14909 NPE testing for RIT; addendum -- forgot import (stack: rev
7f2e09f9c98ebc2b6b738efb13172b0c2c924f6f)
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> NPE testing for RIT
> ---
>
> Key: HBASE-14909
> URL: https://issues.apache.org/jira/browse/HBASE-14909
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14909.branch-1.txt, 14909.txt
>
>
> Master build just failed because TestMasterFailover failed with NPE testing
> for RIT:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$8.evaluate(HBaseTestingUtility.java:3770)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:191)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:143)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:3737)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilNoRegionsInTransition(HBaseTestingUtility.java:3843)
> at
> org.apache.hadoop.hbase.master.TestMasterFailover.testMetaInTransitionWhenMasterFailover(TestMasterFailover.java:347)
> {code}
> Small fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036936#comment-15036936
]
Hudson commented on HBASE-14819:
SUCCESS: Integrated in HBase-1.3 #411 (See
[https://builds.apache.org/job/HBase-1.3/411/])
HBASE-14819 hbase-it tests failing with OOME; permgen (stack: rev
f730a375e3c67b8e3bbfa2ec4e4488f35bd2db0d)
* hbase-it/pom.xml
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.add.maxpermsize.patch, 14819.addendum.patch,
> 14819.branch-1.txt, 14819.total.master.changes.patch, 14819v2.txt, Screen
> Shot 2015-11-16 at 11.37.41 PM.png, itag.txt, more.heap.patch
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
> org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43)
> at
>
[jira] [Commented] (HBASE-14908) TestRowCounter flakey especially on branch-1
[ https://issues.apache.org/jira/browse/HBASE-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036935#comment-15036935 ] Hudson commented on HBASE-14908: SUCCESS: Integrated in HBase-1.3 #411 (See [https://builds.apache.org/job/HBase-1.3/411/]) HBASE-14908 TestRowCounter flakey especially on branch-1 (stack: rev 81de832732ffabf30add0251fb2690470cb41524) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java > TestRowCounter flakey especially on branch-1 > > > Key: HBASE-14908 > URL: https://issues.apache.org/jira/browse/HBASE-14908 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Affects Versions: 1.2.0 >Reporter: stack >Assignee: stack > Attachments: 14908.patch > > > It fails a bunch, about one in five runs: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/417/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterTimeRange/history/ > Looking at log it seems to be making good progress without issue but the test > is running four mapreduce tasks and it wants to finish inside three minutes > (medium sized test). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14746) Run less client threads in tests
[ https://issues.apache.org/jira/browse/HBASE-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14746: -- Attachment: lessthreads (1).txt Retry > Run less client threads in tests > > > Key: HBASE-14746 > URL: https://issues.apache.org/jira/browse/HBASE-14746 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Attachments: lessthreads (1).txt, lessthreads.txt, lessthreads.txt > > > Looking at thread counts reported by the nice 'after:' log message, I see the > likes of TestReplicationKillSlaveRS and all of its derivatives reporting > hundreds of threads (shows as over 1000 in one run here). > Running the test standalone, I see the above run up the threads to 700 > (only!). Need to figure why the discrepancy. > The attached patch makes some difference -- cutting down the client thread > count -- but not enough. Let me look more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14888) ClusterSchema: Add Namespace Operations
[ https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036930#comment-15036930 ] stack commented on HBASE-14888: --- So hang is in here in: testNotPullingDeadRegionServerFromZK in TestMasterNoCluster Our namespace setup is hanging because no hbase:meta assigned (intentionally done in this mock). Looking at build without patch, we skirt hbase:meta deploy and by-pass namespace setup. My patch does not skirt namespace setup. Let me add that to this mocked up master. > ClusterSchema: Add Namespace Operations > --- > > Key: HBASE-14888 > URL: https://issues.apache.org/jira/browse/HBASE-14888 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: > 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch, > 14888.patch, 14888v3.txt, 14888v4.txt, 14888v5.txt > > > Add in a ClusterSchema Interface. It will have all API for all cluster > manipulation; adding namespaces, tables, amending column families, etc. The > idea is to gather up our mess and put it all behind a tidy API that all works > the same way whatever you changing returning a Future to wait on and behind > the scenes driving Procedures. > This patch does namespace operations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14719) Add metric for number of MasterProcWALs
[ https://issues.apache.org/jira/browse/HBASE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036933#comment-15036933 ] Vrishal Kulkarni commented on HBASE-14719: -- Then for simplicity sake, I'll go back to having this metric in the master. I'll repost with that code change. > Add metric for number of MasterProcWALs > --- > > Key: HBASE-14719 > URL: https://issues.apache.org/jira/browse/HBASE-14719 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Vrishal Kulkarni > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14719.1.patch, HBASE-14719.2.patch, > HBASE-14719.3.patch, HBASE-14719.patch > > > Lets add monitoring to this so that we can see when it starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14719) Add metric for number of MasterProcWALs
[ https://issues.apache.org/jira/browse/HBASE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036941#comment-15036941 ] Elliott Clark commented on HBASE-14719: --- No I think this is fine. Let me commit it. > Add metric for number of MasterProcWALs > --- > > Key: HBASE-14719 > URL: https://issues.apache.org/jira/browse/HBASE-14719 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Vrishal Kulkarni > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14719.1.patch, HBASE-14719.2.patch, > HBASE-14719.3.patch, HBASE-14719.patch > > > Lets add monitoring to this so that we can see when it starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14888) ClusterSchema: Add Namespace Operations
[ https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14888: -- Attachment: 14888v6.txt Fix hanging test and address checkstyle issues. > ClusterSchema: Add Namespace Operations > --- > > Key: HBASE-14888 > URL: https://issues.apache.org/jira/browse/HBASE-14888 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: > 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch, > 14888.patch, 14888v3.txt, 14888v4.txt, 14888v5.txt, 14888v6.txt > > > Add in a ClusterSchema Interface. It will have all API for all cluster > manipulation; adding namespaces, tables, amending column families, etc. The > idea is to gather up our mess and put it all behind a tidy API that all works > the same way whatever you changing returning a Future to wait on and behind > the scenes driving Procedures. > This patch does namespace operations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13857) Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
[
https://issues.apache.org/jira/browse/HBASE-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrishal Kulkarni updated HBASE-13857:
-
Status: Patch Available (was: Open)
Patch uploaded for review
> Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
> -
>
> Key: HBASE-13857
> URL: https://issues.apache.org/jira/browse/HBASE-13857
> Project: HBase
> Issue Type: Bug
> Components: regionserver, UI
>Affects Versions: 0.98.0
>Reporter: Lars George
>Assignee: Vrishal Kulkarni
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-13857.patch
>
>
> The template has this:
> {noformat}
>
> ...
> Slow WAL Append Count
>
>
>
> <% 0 %>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14901) There is duplicated code to create/manage encryption keys
[
https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036984#comment-15036984
]
Hadoop QA commented on HBASE-14901:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775384/HBASE-14901.3.patch
against master branch at commit da0cc598feab995eed12527d90805dd627674035.
ATTACHMENT ID: 12775384
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16733//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16733//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16733//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16733//console
This message is automatically generated.
> There is duplicated code to create/manage encryption keys
> -
>
> Key: HBASE-14901
> URL: https://issues.apache.org/jira/browse/HBASE-14901
> Project: HBase
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.0.0
>Reporter: Nate Edel
>Assignee: Nate Edel
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch,
> HBASE-14901.3.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> There is duplicated code from MobUtils.createEncryptionContext in HStore, and
> there is a subset of that code in HFileReaderImpl.
> Refactored key selection
> Moved both to EncryptionUtil.java
> Can't figure out how to write a unit test for this, but there's no new code
> just refactoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14906) Improvements on FlushLargeStoresPolicy
[
https://issues.apache.org/jira/browse/HBASE-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037148#comment-15037148
]
Duo Zhang commented on HBASE-14906:
---
{quote}
About config in table description, I think we need to preserve a way for
experts to override the default value, in case they really need to set a
smaller value than region.getMemstoreFlushSize() / familyNumber for their use
case
{quote}
Sounds reasonable. What about introduce two configurations here? One is the
lower bound of lower bound, one is the exact lower bound? Make global config
and table config have different meanings is a little confusing I think.
> Improvements on FlushLargeStoresPolicy
> --
>
> Key: HBASE-14906
> URL: https://issues.apache.org/jira/browse/HBASE-14906
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14906.patch
>
>
> When checking FlushLargeStoragePolicy, found below possible improving points:
> 1. Currently in selectStoresToFlush, we will do the selection no matter how
> many actual families, which is not necessary for one single family
> 2. Default value for hbase.hregion.percolumnfamilyflush.size.lower.bound
> could not fit in all cases, and requires user to know details of the
> implementation to properly set it. We propose to use
> "hbase.hregion.memstore.flush.size/column_family_number" instead:
> {noformat}
>
> hbase.hregion.percolumnfamilyflush.size.lower.bound
> 16777216
>
> If FlushLargeStoresPolicy is used and there are multiple column families,
> then every time that we hit the total memstore limit, we find out all the
> column families whose memstores exceed a "lower bound" and only flush them
> while retaining the others in memory. The "lower bound" will be
> "hbase.hregion.memstore.flush.size / column_family_number" by default
> unless value of this property is larger than that. If none of the families
> have their memstore size more than lower bound, all the memstores will be
> flushed (just as usual).
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14746) Run less client threads in tests
[ https://issues.apache.org/jira/browse/HBASE-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14746: -- Resolution: Later Status: Resolved (was: Patch Available) Resolving as later. Small potatoes. > Run less client threads in tests > > > Key: HBASE-14746 > URL: https://issues.apache.org/jira/browse/HBASE-14746 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Attachments: lessthreads (1).txt, lessthreads.txt, lessthreads.txt > > > Looking at thread counts reported by the nice 'after:' log message, I see the > likes of TestReplicationKillSlaveRS and all of its derivatives reporting > hundreds of threads (shows as over 1000 in one run here). > Running the test standalone, I see the above run up the threads to 700 > (only!). Need to figure why the discrepancy. > The attached patch makes some difference -- cutting down the client thread > count -- but not enough. Let me look more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14873) Problems around BoundedByteBufferPool providing direct buffers
[
https://issues.apache.org/jira/browse/HBASE-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037212#comment-15037212
]
ramkrishna.s.vasudevan commented on HBASE-14873:
Interesting and useful information here.
bq.DirectBuffer's override method calls Unsafe's copyMemory under the limit
Bits.UNSAFE_COPY_THRESHOLD, which has been bq.introduced for scaling well,
according to google. How copyFromBufferToArray can omit the logic? Another
thing I should bq.mention is that copyFromBufferToArray is forced to use
one-by-one copy for the other VM implementations, which is an almost bq.bug.
Yes true. But our intention was to mainly avoid all those boundary checks that
Oracle's impl has. Some of the earlier tests suggested that the NIO buffer APIs
were not allowing the JVM to do proper inlining because of the boundary checks
in every API it has. the same is true with the onheap BBs also. The aim of
these Byte and BytebufferUtils was to avoid these and directly go with Unsafe.
One thing to see is that if all the are all onheap we just treat them as Arrays
But the problems you seem to say needs to handled in the BBUtils impl? If so,
that seems to be an important thing. But our code handles the endian thing -
then may be that is not enough.
bq.In addition, Oracle seems to plan to remove Unsafe in Java 9.
This as we know they may remove but I think they are planning to provide some
other way of using them? There was a study internally conducted and it was
found that many of the projects (apache projects included) make use of these
DBBs and Unsafe Utils quite extensively.
> Problems around BoundedByteBufferPool providing direct buffers
> --
>
> Key: HBASE-14873
> URL: https://issues.apache.org/jira/browse/HBASE-14873
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: HBASE-14873-V2.patch, HBASE-14873.patch,
> HBASE-14873.patch, HBASE-14873.patch
>
>
> HBASE-13819 made BoundedByteBufferPool provide direct buffers.
> See RpcServer.java:
> {code}
> ...
> class Call implements RpcCallContext {
> protected synchronized void setResponse(...) {
> ...
> this.cellBlock = ipcUtil.buildCellBlock(..., reservoir);
> ...
> bc = new BufferChain(..., this.cellBlock);
> if (connection.useWrap) {
> bc = wrapWithSasl(bc);
> }
> ...
> private BufferChain wrapWithSasl(BufferChain bc) throws IOException {
> ...
> byte[] responseBytes = bc.getBytes();
> ...
> {code}
> {{cellBlock}} is expected to be a direct buffer retrieved from {{reservoir}}
> (but not always), and {{bc}} may be composed of both direct and non-direct
> buffers.
> And then, see BufferChain.java:
> {code}
> byte [] getBytes() {
> ...
> for (ByteBuffer bb: this.buffers) {
> System.arraycopy(bb.array(), ...);
> {code}
> A direct buffer doesn't give its array, and will throw
> UnsupportedOperationException.
> Another problem; {{cellBlock}} is allowed to be a non-direct buffer, and
> after use it will be put to {{reservoir}}, mixing direct and non-direct
> buffers in the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14906) Improvements on FlushLargeStoresPolicy
[
https://issues.apache.org/jira/browse/HBASE-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-14906:
--
Fix Version/s: 2.0.0
Release Note:
In HBASE-14906 we use "hbase.hregion.memstore.flush.size/column_family_number"
as the default threshold for memstore flush instead of the fixed value through
"hbase.hregion.percolumnfamilyflush.size.lower.bound" property, which makes
the default threshold more flexible to various use case. We also introduce a
new property in name of
"hbase.hregion.percolumnfamilyflush.size.lower.bound.min" with 16M as the
default value to avoid small flush in cases like hundreds of column families.
After this change setting "hbase.hregion.percolumnfamilyflush.size.lower.bound"
in hbase-site.xml won't take effect anymore, but expert users could still set
this property in table descriptor to override the default value just as before
Status: Patch Available (was: Open)
Add release note and submit patch to see what HadoopQA will say
> Improvements on FlushLargeStoresPolicy
> --
>
> Key: HBASE-14906
> URL: https://issues.apache.org/jira/browse/HBASE-14906
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14906.patch, HBASE-14906.v2.patch
>
>
> When checking FlushLargeStoragePolicy, found below possible improving points:
> 1. Currently in selectStoresToFlush, we will do the selection no matter how
> many actual families, which is not necessary for one single family
> 2. Default value for hbase.hregion.percolumnfamilyflush.size.lower.bound
> could not fit in all cases, and requires user to know details of the
> implementation to properly set it. We propose to use
> "hbase.hregion.memstore.flush.size/column_family_number" instead:
> {noformat}
>
> hbase.hregion.percolumnfamilyflush.size.lower.bound
> 16777216
>
> If FlushLargeStoresPolicy is used and there are multiple column families,
> then every time that we hit the total memstore limit, we find out all the
> column families whose memstores exceed a "lower bound" and only flush them
> while retaining the others in memory. The "lower bound" will be
> "hbase.hregion.memstore.flush.size / column_family_number" by default
> unless value of this property is larger than that. If none of the families
> have their memstore size more than lower bound, all the memstores will be
> flushed (just as usual).
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14876) Provide maven archetypes
[ https://issues.apache.org/jira/browse/HBASE-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Vimont updated HBASE-14876: -- Attachment: archetype_prototype02.zip Providing corrected version of the prototype in a new zip file. Fixes following problem: The simple client application generated from the archetype was generating warnings that log4j was not configured properly. This was resolved by copying log4j.properties from the hbase/conf directory into the base* project's src/main/resources directory. *"Base" project refers to project from which the archetype was derived. > Provide maven archetypes > > > Key: HBASE-14876 > URL: https://issues.apache.org/jira/browse/HBASE-14876 > Project: HBase > Issue Type: New Feature > Components: build, Usability >Reporter: Nick Dimiduk >Assignee: Daniel Vimont > Labels: beginner > Attachments: archetype_prototype.zip, archetype_prototype02.zip > > > To help onboard new users, we should provide maven archetypes for hbase > client applications. Off the top of my head, we should have templates for > - hbase client application with all dependencies > - hbase client application using client-shaded jar > - mapreduce application with hbase as input and output (ie, copy table) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037438#comment-15037438
]
Vishal Khandelwal commented on HBASE-14905:
---
[~chenheng] I looked into test patch it covers the scenario where number of
version are different. I would like also like to add test where number of
version are same but data does not in one version. I will add that test upload
a new patch.
Also, in map method we are masking the exception from CompareResult. In logs we
do get BADROW but not exact reason and exeception. I will file separate JIRA
for the same & provide the fix so don't mix 2 things.
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: 2.0.0, 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Fix For: 2.0.0
>
> Attachments: 14905-v2.txt, HBASE-14905.patch, HBASE-14905_v3.patch,
> test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14916) Add checkstyle_report.py to other branches
Appy created HBASE-14916: Summary: Add checkstyle_report.py to other branches Key: HBASE-14916 URL: https://issues.apache.org/jira/browse/HBASE-14916 Project: HBase Issue Type: Bug Reporter: Appy Given test-patch.sh is always run from master, and that it now uses checkstyle_report.py, we should pull back the script to other branches too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14906) Improvements on FlushLargeStoresPolicy
[
https://issues.apache.org/jira/browse/HBASE-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037238#comment-15037238
]
Yu Li commented on HBASE-14906:
---
{quote}
What about introduce two configurations here? One is the lower bound of lower
bound, one is the exact lower bound?
{quote}
Make sense, will rename the global config name to
{{hbase.hregion.percolumnfamilyflush.size.lower.bound.min}} to better reflect
what the property means, or if you have a better name in mind, just let me
know. :-)
This change will invalidate user's previous setting on
{{hbase.hregion.percolumnfamilyflush.size.lower.bound}}, but this would be
by-design since we have a better default ({{region.getMemstoreFlushSize() /
familyNumber}}) now. Will add the note into ReleaseNote.
> Improvements on FlushLargeStoresPolicy
> --
>
> Key: HBASE-14906
> URL: https://issues.apache.org/jira/browse/HBASE-14906
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14906.patch, HBASE-14906.v2.patch
>
>
> When checking FlushLargeStoragePolicy, found below possible improving points:
> 1. Currently in selectStoresToFlush, we will do the selection no matter how
> many actual families, which is not necessary for one single family
> 2. Default value for hbase.hregion.percolumnfamilyflush.size.lower.bound
> could not fit in all cases, and requires user to know details of the
> implementation to properly set it. We propose to use
> "hbase.hregion.memstore.flush.size/column_family_number" instead:
> {noformat}
>
> hbase.hregion.percolumnfamilyflush.size.lower.bound
> 16777216
>
> If FlushLargeStoresPolicy is used and there are multiple column families,
> then every time that we hit the total memstore limit, we find out all the
> column families whose memstores exceed a "lower bound" and only flush them
> while retaining the others in memory. The "lower bound" will be
> "hbase.hregion.memstore.flush.size / column_family_number" by default
> unless value of this property is larger than that. If none of the families
> have their memstore size more than lower bound, all the memstores will be
> flushed (just as usual).
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14906) Improvements on FlushLargeStoresPolicy
[
https://issues.apache.org/jira/browse/HBASE-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-14906:
--
Attachment: HBASE-14906.v2.patch
Update patch according to Duo's comment
> Improvements on FlushLargeStoresPolicy
> --
>
> Key: HBASE-14906
> URL: https://issues.apache.org/jira/browse/HBASE-14906
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14906.patch, HBASE-14906.v2.patch
>
>
> When checking FlushLargeStoragePolicy, found below possible improving points:
> 1. Currently in selectStoresToFlush, we will do the selection no matter how
> many actual families, which is not necessary for one single family
> 2. Default value for hbase.hregion.percolumnfamilyflush.size.lower.bound
> could not fit in all cases, and requires user to know details of the
> implementation to properly set it. We propose to use
> "hbase.hregion.memstore.flush.size/column_family_number" instead:
> {noformat}
>
> hbase.hregion.percolumnfamilyflush.size.lower.bound
> 16777216
>
> If FlushLargeStoresPolicy is used and there are multiple column families,
> then every time that we hit the total memstore limit, we find out all the
> column families whose memstores exceed a "lower bound" and only flush them
> while retaining the others in memory. The "lower bound" will be
> "hbase.hregion.memstore.flush.size / column_family_number" by default
> unless value of this property is larger than that. If none of the families
> have their memstore size more than lower bound, all the memstores will be
> flushed (just as usual).
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13857) Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
[
https://issues.apache.org/jira/browse/HBASE-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037269#comment-15037269
]
Hadoop QA commented on HBASE-13857:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774890/HBASE-13857.patch
against master branch at commit da0cc598feab995eed12527d90805dd627674035.
ATTACHMENT ID: 12774890
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16737//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16737//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16737//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16737//console
This message is automatically generated.
> Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
> -
>
> Key: HBASE-13857
> URL: https://issues.apache.org/jira/browse/HBASE-13857
> Project: HBase
> Issue Type: Bug
> Components: regionserver, UI
>Affects Versions: 0.98.0
>Reporter: Lars George
>Assignee: Vrishal Kulkarni
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-13857.patch
>
>
> The template has this:
> {noformat}
>
> ...
> Slow WAL Append Count
>
>
>
> <% 0 %>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14916) Add checkstyle_report.py to other branches
[ https://issues.apache.org/jira/browse/HBASE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14916: - Description: Given test-patch.sh is always run from master, and that it now uses checkstyle_report.py, we should pull back the script to other branches too. Otherwise we see error like: /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/jenkins.build/dev-support/test-patch.sh: line 662: /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/dev-support/checkstyle_report.py: No such file or directory [reference|https://builds.apache.org/job/PreCommit-HBASE-Build/16734//consoleFull] was:Given test-patch.sh is always run from master, and that it now uses checkstyle_report.py, we should pull back the script to other branches too. > Add checkstyle_report.py to other branches > -- > > Key: HBASE-14916 > URL: https://issues.apache.org/jira/browse/HBASE-14916 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14916-branch-1.patch > > > Given test-patch.sh is always run from master, and that it now uses > checkstyle_report.py, we should pull back the script to other branches too. > Otherwise we see error like: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/jenkins.build/dev-support/test-patch.sh: > line 662: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/dev-support/checkstyle_report.py: > No such file or directory > [reference|https://builds.apache.org/job/PreCommit-HBASE-Build/16734//consoleFull] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14420) Zombie Stomping Session
[ https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14420: -- Attachment: none_fix.txt Retry (I'd broken test-patch.sh) > Zombie Stomping Session > --- > > Key: HBASE-14420 > URL: https://issues.apache.org/jira/browse/HBASE-14420 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: hangers.txt, none_fix (1).txt, none_fix.txt, > none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, > none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, > none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, > none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, > none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, > none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, > none_fix.txt > > > Patch build are now failing most of the time because we are dropping zombies. > I confirm we are doing this on non-apache build boxes too. > Left-over zombies consume resources on build boxes (OOME cannot create native > threads). Having to do multiple test runs in the hope that we can get a > non-zombie-making build or making (arbitrary) rulings that the zombies are > 'not related' is a productivity sink. And so on... > This is an umbrella issue for a zombie stomping session that started earlier > this week. Will hang sub-issues of this one. Am running builds back-to-back > on little cluster to turn out the monsters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14873) Problems around BoundedByteBufferPool providing direct buffers
[
https://issues.apache.org/jira/browse/HBASE-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037274#comment-15037274
]
Anoop Sam John commented on HBASE-14873:
In case of on heap buffer and byte arrays , for reading int/long etc we have
made it to Unsafe way for perf reasons. I think this was added an year back or
so. We can avoid java way of reading byte one after the other and calc the
int/long , which we were doing before. The Unsafe way seems to be much much
fast.
'Alignment ' - Will do more checking on this. This might be an issue from 0.94
onwards if it turns out to be.
> Problems around BoundedByteBufferPool providing direct buffers
> --
>
> Key: HBASE-14873
> URL: https://issues.apache.org/jira/browse/HBASE-14873
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: HBASE-14873-V2.patch, HBASE-14873.patch,
> HBASE-14873.patch, HBASE-14873.patch
>
>
> HBASE-13819 made BoundedByteBufferPool provide direct buffers.
> See RpcServer.java:
> {code}
> ...
> class Call implements RpcCallContext {
> protected synchronized void setResponse(...) {
> ...
> this.cellBlock = ipcUtil.buildCellBlock(..., reservoir);
> ...
> bc = new BufferChain(..., this.cellBlock);
> if (connection.useWrap) {
> bc = wrapWithSasl(bc);
> }
> ...
> private BufferChain wrapWithSasl(BufferChain bc) throws IOException {
> ...
> byte[] responseBytes = bc.getBytes();
> ...
> {code}
> {{cellBlock}} is expected to be a direct buffer retrieved from {{reservoir}}
> (but not always), and {{bc}} may be composed of both direct and non-direct
> buffers.
> And then, see BufferChain.java:
> {code}
> byte [] getBytes() {
> ...
> for (ByteBuffer bb: this.buffers) {
> System.arraycopy(bb.array(), ...);
> {code}
> A direct buffer doesn't give its array, and will throw
> UnsupportedOperationException.
> Another problem; {{cellBlock}} is allowed to be a non-direct buffer, and
> after use it will be put to {{reservoir}}, mixing direct and non-direct
> buffers in the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14873) Problems around BoundedByteBufferPool providing direct buffers
[
https://issues.apache.org/jira/browse/HBASE-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037156#comment-15037156
]
Hiroshi Ikeda commented on HBASE-14873:
---
bq. Why sir?
Sorry for my poor expression, I just complained that I prefer no waste.
As for Unsafe, this is very subtle. I just found ByteBufferUtils uses
Unsafe.getInt etc. with no restriction, but the Oracle implementation uses
these methods only if the class variable {{unaligned}} (commented as "Cached
unaligned-access capability") is true, which seems to be calculated whether the
architecture is i386, x86, amd64, or x86_64. ByteBufferUtils also uses Unsafe
for on-heap buffers, but Unsafe uses native methods and it consequently
prevents VM optimization, and it might eventually wastes for on-heap buffers in
long run. In addition, Oracle seems to plan to remove Unsafe in Java 9.
{quote}
bq. In other words there are situations where we should intentionally use the
right buffer in the right place.
Makes sense. I think the places where BBBP have been deployed so far make sense.
{quote}
I mean, it doesn't make sense that users can switch direct/non-direct buffers
to pool (in the 1.3 branch) and developers hesitate to improve code to take
advantage of direct/non-direct buffers.
bq. This issue that you filed is a dirty bug we need to fix regardless, right?
Yes, this is an absolutely serious bug that is always occurred when wrapping a
cell block, and that must be fixed in any way, regardless of my complaining :(
> Problems around BoundedByteBufferPool providing direct buffers
> --
>
> Key: HBASE-14873
> URL: https://issues.apache.org/jira/browse/HBASE-14873
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: HBASE-14873-V2.patch, HBASE-14873.patch,
> HBASE-14873.patch, HBASE-14873.patch
>
>
> HBASE-13819 made BoundedByteBufferPool provide direct buffers.
> See RpcServer.java:
> {code}
> ...
> class Call implements RpcCallContext {
> protected synchronized void setResponse(...) {
> ...
> this.cellBlock = ipcUtil.buildCellBlock(..., reservoir);
> ...
> bc = new BufferChain(..., this.cellBlock);
> if (connection.useWrap) {
> bc = wrapWithSasl(bc);
> }
> ...
> private BufferChain wrapWithSasl(BufferChain bc) throws IOException {
> ...
> byte[] responseBytes = bc.getBytes();
> ...
> {code}
> {{cellBlock}} is expected to be a direct buffer retrieved from {{reservoir}}
> (but not always), and {{bc}} may be composed of both direct and non-direct
> buffers.
> And then, see BufferChain.java:
> {code}
> byte [] getBytes() {
> ...
> for (ByteBuffer bb: this.buffers) {
> System.arraycopy(bb.array(), ...);
> {code}
> A direct buffer doesn't give its array, and will throw
> UnsupportedOperationException.
> Another problem; {{cellBlock}} is allowed to be a non-direct buffer, and
> after use it will be put to {{reservoir}}, mixing direct and non-direct
> buffers in the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037190#comment-15037190
]
Hadoop QA commented on HBASE-14905:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775430/14905-v2.txt
against master branch at commit da0cc598feab995eed12527d90805dd627674035.
ATTACHMENT ID: 12775430
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16735//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16735//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16735//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16735//console
This message is automatically generated.
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: 2.0.0, 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Fix For: 2.0.0
>
> Attachments: 14905-v2.txt, HBASE-14905.patch, HBASE-14905_v3.patch,
> test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037292#comment-15037292 ] Heng Chen commented on HBASE-14895: --- I see.. It seems StoreScanner has a great change in HBASE-13082 > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14916) Add checkstyle_report.py to other branches
[ https://issues.apache.org/jira/browse/HBASE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14916: - Attachment: HBASE-14916-branch-1.patch > Add checkstyle_report.py to other branches > -- > > Key: HBASE-14916 > URL: https://issues.apache.org/jira/browse/HBASE-14916 > Project: HBase > Issue Type: Bug >Reporter: Appy > Attachments: HBASE-14916-branch-1.patch > > > Given test-patch.sh is always run from master, and that it now uses > checkstyle_report.py, we should pull back the script to other branches too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14888) ClusterSchema: Add Namespace Operations
[
https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037265#comment-15037265
]
Hadoop QA commented on HBASE-14888:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775436/14888v6.txt
against master branch at commit da0cc598feab995eed12527d90805dd627674035.
ATTACHMENT ID: 12775436
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16736//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16736//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16736//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16736//console
This message is automatically generated.
> ClusterSchema: Add Namespace Operations
> ---
>
> Key: HBASE-14888
> URL: https://issues.apache.org/jira/browse/HBASE-14888
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments:
> 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch,
> 14888.patch, 14888v3.txt, 14888v4.txt, 14888v5.txt, 14888v6.txt
>
>
> Add in a ClusterSchema Interface. It will have all API for all cluster
> manipulation; adding namespaces, tables, amending column families, etc. The
> idea is to gather up our mess and put it all behind a tidy API that all works
> the same way whatever you changing returning a Future to wait on and behind
> the scenes driving Procedures.
> This patch does namespace operations first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
