date:20121227


[ 
https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539866#comment-13539866
 ] 

stack commented on HBASE-7188:
--

Elliott, now client-side, when Invoker does getProtocolVersion, it is reading 
the local version of the protocol (HBaseClientRPC.getProtocolVersion).  Is that 
right?  When does client ever learn the server version?  When I set up the 
protocol, should I not be going to the server to ask its version of the 
protocol?  At least for the server version of this protocol?  Or how do you see 
this versioning and proxying working on protocol Interfaces now?  Previous, 
after getting proxy, we used to do a getProtocolVersion and throw exception if 
the client and server versions didn't match.  We don't do that anymore it 
seems.  Here is old code from 0.94:

{code}
 17   /** Construct a client-side proxy object that implements the named 
protocol,
 16* talking to a server at the named address. */
 15   public VersionedProtocol getProxy(
 14   Class? extends VersionedProtocol protocol, long clientVersion,
 13   InetSocketAddress addr, User ticket,
 12   Configuration conf, SocketFactory factory, int rpcTimeout)
 11 throws IOException {
 10 
  9   VersionedProtocol proxy =
  8   (VersionedProtocol) Proxy.newProxyInstance(
  7   protocol.getClassLoader(), new Class[] { protocol },
  6   new Invoker(protocol, addr, ticket, conf, factory, 
rpcTimeout));
  5 if (proxy instanceof VersionedProtocol) {
  4   long serverVersion = ((VersionedProtocol)proxy)
  3 .getProtocolVersion(protocol.getName(), clientVersion);
  2   if (serverVersion != clientVersion) {
  1 throw new HBaseRPC.VersionMismatch(protocol.getName(), 
clientVersion,
  0   serverVersion);   


  1   }
  2 }
  3 return proxy;
  4   }
{code}

 Move classes into hbase-client
 --

 Key: HBASE-7188
 URL: https://issues.apache.org/jira/browse/HBASE-7188
 Project: HBase
  Issue Type: Sub-task
  Components: Client, IPC/RPC
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 0.96.0

 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7390) Add extra test cases for assignement on the region server and fix the related issues


[ 
https://issues.apache.org/jira/browse/HBASE-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539867#comment-13539867
 ] 

nkeywal commented on HBASE-7390:


It's on a review board, there it a +1 from Nick there.
Is there anyone else reviewing it? If not, I will commit next Monday.

 Add extra test cases for assignement on the region server and fix the related 
 issues
 

 Key: HBASE-7390
 URL: https://issues.apache.org/jira/browse/HBASE-7390
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment, regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
 Fix For: 0.96.0

 Attachments: 7390.v1.patch, 7390.v2.patch, 7390.v3.patch, 
 7390.v4.patch, 7390.v6.patch, 7390.v7.patch, 7390.v8.patch, 
 assignment_zk_states.jpg


 We don't have a lot of tests on the region server itself.
 Here are some.
 Some of them are failing, feedback welcome.
 See as well the attached state diagram for the ZK nodes on assignment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment

2012-12-27 Thread shen guanpu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539873#comment-13539873
 ] 

shen guanpu commented on HBASE-7263:


HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned, is 
that you have to wait for updates on other rows, since MVCC is per-row.

Do you mean it is not per-row for option 2( Have an MVCC per-row (table 
configuration): this avoids the unnecessary contention of 1))

 Investigate more fine grained locking for checkAndPut/append/increment
 --

 Key: HBASE-7263
 URL: https://issues.apache.org/jira/browse/HBASE-7263
 Project: HBase
  Issue Type: Improvement
  Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor

 HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
 {quote}
 1) Waiting for the MVCC to advance for read/updates: the downside is that you 
 have to wait for updates on other rows.
 2) Have an MVCC per-row (table configuration): this avoids the unnecessary 
 contention of 1)
 3) Transform the read/updates to write-only with rollup on read.. E.g. an 
 increment would just have the number of values to increment.
 {quote}
 HBASE-7051 and HBASE-4583 implement option #1.  The downside, as mentioned, 
 is that you have to wait for updates on other rows, since MVCC is per-row.
 Another option occurred to me that I think is worth investigating: rely on a 
 row-level read/write lock rather than MVCC.
 Here is pseudo-code for what exists today for read/updates like checkAndPut
 {code}
 (1)  Acquire RowLock
 (1a) BeginMVCC + Finish MVCC
 (2)  Begin MVCC
 (3)  Do work
 (4)  Release RowLock
 (5)  Append to WAL
 (6)  Finish MVCC
 {code}
 Write-only operations (e.g. puts) are the same, just without step 1a.
 Now, consider the following instead:
 {code}
 (1)  Acquire RowLock
 (1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
 (1b) Grab RowReadLock (new step!)
 (2)  Begin MVCC
 (3)  Do work
 (4)  Release RowLock
 (5)  Append to WAL
 (6)  Finish MVCC
 (7)  Release RowReadLock (new step!)
 {code}
 As before, write-only operations are the same, just without step 1a.
 The difference here is that writes grab a row-level read lock and hold it 
 until the MVCC is completed.  The nice property that this gives you is that 
 read/updates can tell when the MVCC is done on a per-row basis, because they 
 can just try to acquire the write-lock which will block until the MVCC is 
 competed for that row in step 7.
 There is overhead for acquiring the read lock that I need to measure, but it 
 should be small, since there will never be any blocking on acquiring the 
 row-level read lock.  This is because the read lock can only block if someone 
 else holds the write lock, but both the write and read lock are only acquired 
 under the row lock.
 I ran a quick test of this approach over a region (this directly interacts 
 with HRegion, so no client effects):
 - 30 threads
 - 5000 increments per thread
 - 30 columns per increment
 - Each increment uniformly distributed over 500,000 rows
 - 5 trials
 Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
 Today: 13950 ms
 The locking approach: 10877 ms
 So it looks like an improvement, at least wrt increment.  As mentioned, I 
 need to measure the overhead of acquiring the read lock for puts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable

Liu Shaohui created HBASE-7441:
--

 Summary: Make ClusterManager in IntegrationTestingUtility pluggable
 Key: HBASE-7441
 URL: https://issues.apache.org/jira/browse/HBASE-7441
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Priority: Minor


After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster.
The ClusterManager use ssh to stop/start the rs or master without passwd. To 
support other cluster manager tool, we need to make clusterManager in 
IntegrationTestingUtility pluggable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable


 [ 
https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-7441:
---

Status: Open  (was: Patch Available)

 Make ClusterManager in IntegrationTestingUtility pluggable
 --

 Key: HBASE-7441
 URL: https://issues.apache.org/jira/browse/HBASE-7441
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Priority: Minor
  Labels: patch
 Fix For: 0.94.4

 Attachments: HBASE-7441-0.94-v1.patch

   Original Estimate: 3h
  Remaining Estimate: 3h

 After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster.
 The ClusterManager use ssh to stop/start the rs or master without passwd. To 
 support other cluster manager tool, we need to make clusterManager in 
 IntegrationTestingUtility pluggable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable


 [ 
https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-7441:
---

Labels: newbie patch  (was: patch)
Status: Patch Available  (was: Open)

 Make ClusterManager in IntegrationTestingUtility pluggable
 --

 Key: HBASE-7441
 URL: https://issues.apache.org/jira/browse/HBASE-7441
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Priority: Minor
  Labels: patch, newbie
 Fix For: 0.94.4

 Attachments: HBASE-7441-0.94-v1.patch

   Original Estimate: 3h
  Remaining Estimate: 3h

 After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster.
 The ClusterManager use ssh to stop/start the rs or master without passwd. To 
 support other cluster manager tool, we need to make clusterManager in 
 IntegrationTestingUtility pluggable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable


 [ 
https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-7441:
---

Fix Version/s: 0.94.4
   Labels: patch  (was: )
   Status: Patch Available  (was: Open)

 Make ClusterManager in IntegrationTestingUtility pluggable
 --

 Key: HBASE-7441
 URL: https://issues.apache.org/jira/browse/HBASE-7441
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Priority: Minor
  Labels: patch
 Fix For: 0.94.4

 Attachments: HBASE-7441-0.94-v1.patch

   Original Estimate: 3h
  Remaining Estimate: 3h

 After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster.
 The ClusterManager use ssh to stop/start the rs or master without passwd. To 
 support other cluster manager tool, we need to make clusterManager in 
 IntegrationTestingUtility pluggable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable


 [ 
https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-7441:
---

Attachment: HBASE-7441-0.94-v1.patch

 Make ClusterManager in IntegrationTestingUtility pluggable
 --

 Key: HBASE-7441
 URL: https://issues.apache.org/jira/browse/HBASE-7441
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Priority: Minor
  Labels: patch
 Fix For: 0.94.4

 Attachments: HBASE-7441-0.94-v1.patch

   Original Estimate: 3h
  Remaining Estimate: 3h

 After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster.
 The ClusterManager use ssh to stop/start the rs or master without passwd. To 
 support other cluster manager tool, we need to make clusterManager in 
 IntegrationTestingUtility pluggable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable


[ 
https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539905#comment-13539905
 ] 

Liu Shaohui commented on HBASE-7441:


Could you give some suggestion? [~sershe]

 Make ClusterManager in IntegrationTestingUtility pluggable
 --

 Key: HBASE-7441
 URL: https://issues.apache.org/jira/browse/HBASE-7441
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Priority: Minor
  Labels: newbie, patch
 Fix For: 0.94.4

 Attachments: HBASE-7441-0.94-v1.patch

   Original Estimate: 3h
  Remaining Estimate: 3h

 After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster.
 The ClusterManager use ssh to stop/start the rs or master without passwd. To 
 support other cluster manager tool, we need to make clusterManager in 
 IntegrationTestingUtility pluggable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7440) ReplicationZookeeper#addPeer is racy


[ 
https://issues.apache.org/jira/browse/HBASE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539912#comment-13539912
 ] 

Matteo Bertozzi commented on HBASE-7440:


If you use createAndWatch() the node is not watched if it already exists,
instead if you use createNodeIfNotExistsAndWatch(), as suggested by Lars, you 
get the watcher on the newly created one or on the old one.
I think that you want the second behaviour, right?

What do you expect if the node already exists, and is set to DISABLED?
I think is fine in that state, right? so, you don't need to check it or force 
it to ENABLED.

 ReplicationZookeeper#addPeer is racy
 

 Key: HBASE-7440
 URL: https://issues.apache.org/jira/browse/HBASE-7440
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.3
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
 Fix For: 0.96.0, 0.94.4

 Attachments: HBASE-7440-v0.patch, HBASE-7440-v1.patch


 While adding a peer, ReplicationZK does the znodes creation in three 
 transactions. Create :
 a) peers znode
 b) peerId specific znode, and
 c) peerState znode
 There is a PeerWatcher which invokes getPeer() (after steps b) and c)). If it 
 happens that while adding a peer, the control flows to getPeer() and step c) 
 has not been processed, it may results in a state where the peer will not be 
 added. This happens while running 
 TestMasterReplication#testCyclicReplication().
 {code}
 2012-12-26 07:36:35,187 INFO  
 [RegionServer:0;p0120.X,38423,1356536179470-EventThread] 
 zookeeper.RecoverableZooKeeper(447): Node /2/replication/peers/1/peer-state 
 already exists and this is not a retry
 2012-12-26 07:36:35,188 ERROR 
 [RegionServer:0;p0120.X,38423,1356536179470-EventThread] 
 regionserver.ReplicationSourceManager$PeersWatcher(527): Error while adding a 
 new peer
 org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
 NodeExists for /2/replication/peers/1/peer-state
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:428)
   at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:410)
   at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:1044)
   at 
 org.apache.hadoop.hbase.replication.ReplicationPeer.startStateTracker(ReplicationPeer.java:82)
   at 
 org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:344)
   at 
 org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:307)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:519)
   at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:315)
   at 
 org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
 2012-12-26 07:36:35,188 DEBUG 
 [RegionServer:0;p0120.X,55742,1356536171947-EventThread] 
 zookeeper.ZKUtil(1545): regionserver:55742-0x13bd7db39580004 Retrieved 36 
 byte(s) of data from znode /1/hbaseid; data=9ce66123-d3e8-4ae9-a249-afe03...
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7294) Check for snapshot file cleaners on start


 [ 
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-7294:
---

Attachment: HBASE-7294-v6.patch

 * Changed exception message using hbase.snapshot.enabled instead of the 
cleaners
 * Added warn on cleaners enabled, but snapshot.enabled false/not set
 * Added a test to verify different conf settings and expected result

 Check for snapshot file cleaners on start
 -

 Key: HBASE-7294
 URL: https://issues.apache.org/jira/browse/HBASE-7294
 Project: HBase
  Issue Type: Sub-task
  Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
 Fix For: hbase-6055, 0.96.0

 Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch, 
 HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch, 
 HBASE-7294-v6.patch


 Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to 
 ensure that any hfiles or hlogs (respectively) that are currently part of a 
 snapshot are not removed from their respective archive directories (.archive 
 and .oldlogs).
 From Matteo Bertozzi:
 {quote}
 currently the snapshot cleaner is not in hbase-default.xml
 and there's no warning/exception on snapshot/restore operation, if not 
 enabled.
 even if we add the cleaner to the hbase-default.xml how do we ensure that the 
 user doesn't remove it?
 Do we want to hardcode the cleaner at master startup?
 Do we want to add a check in snapshot/restore that throws an exception if the 
 cleaner is not enabled?
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7419) revisit hfilelink file name format.


[ 
https://issues.apache.org/jira/browse/HBASE-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539954#comment-13539954
 ] 

Matteo Bertozzi commented on HBASE-7419:


I agree, the introduction of a special character simplifies the 
identification of a HFileLink, and allows to remove the FileNotFoundException 
handling from the StoreFile.

 revisit hfilelink file name format.
 ---

 Key: HBASE-7419
 URL: https://issues.apache.org/jira/browse/HBASE-7419
 Project: HBase
  Issue Type: Sub-task
  Components: Client, master, regionserver, snapshots, Zookeeper
Reporter: Jonathan Hsieh
 Fix For: hbase-6055, 0.96.0


 Valid table names are concatted with a '.' to a valid regions names is also a 
 valid table name, and lead to the incorrect interpretation.
 {code}
 true hfile name constraints: [0-9]+(?:_SeqID_[0-9]+)?
 region name constraints: [a-f0-9]{16}  (but we currently just use 
 [a-f0-9]+.)
 table name constraints : [a-zA-Z0-9_][a-zA-Z0-9_.-]*
 {code}
 Notice that the table name constraints completely covers all region name 
 constraints and true hfile name constraints.   (a valid hfile name is a valid 
 part of a table name, and a valid enc region name is a valid part of a table 
 name.
 Currently the hfilelink filename convention is hfile-region-table.  
 Unfortunately, making a ref to this uses the name 
 hfile-region-table.parentregion -- the contactnation of 
 table.parentregion is a valid table name used to get interpreted as such. 
  The fix in HBASE-7339 requires a FileNotFoundException before going down the 
 hfile link resolution path. 
 Regardless of what we do, we need to add some char invalid for table names to 
 the hfilelink or reference filename convention.
 Suggestion: if we changed the order of the hfile-link name we could avoid 
 some of the confusion -- table@region-hfile.parentregion (or some 
 other separator char than '@') could be used to avoid handling on the initial 
 filenotfoundexception but I think we'd still need a good chunk of the logic 
 to handle opening half-storefile reader throw a hfilelink.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539963#comment-13539963
 ] 

Matteo Bertozzi commented on HBASE-7400:


changes between hadoop 1 and 2.
and since in the test, the mappers seems running in the same jvm
fs.close() called by the mapper interferes with the one used by the main().

 ExportSnapshot mapper closes the FileSystem
 ---

 Key: HBASE-7400
 URL: https://issues.apache.org/jira/browse/HBASE-7400
 Project: HBase
  Issue Type: Sub-task
  Components: Client, master, regionserver, snapshots, Zookeeper
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Blocker
 Fix For: hbase-6055, 0.96.0

 Attachments: HBASE-7400-v0.patch


 The ExportSnapshotMapper closes the FileSystem on cleanup,
 and in case of shared instead, since there's no ref count, the job fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7188) Move classes into hbase-client

2012-12-27 Thread Elliott Clark (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539962#comment-13539962
 ] 

Elliott Clark commented on HBASE-7188:
--

[~stack] It seems like that's been changed for quite a while (Here's the file 
before we started splitting this out.): 
https://github.com/apache/hbase/blob/fd7647b25d316fef32413b3caeddcf6ba32485f9/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcEngine.java#L112
  From a quick skim it looks like HBASE-5705 was the jira that changed that 
code.

 Move classes into hbase-client
 --

 Key: HBASE-7188
 URL: https://issues.apache.org/jira/browse/HBASE-7188
 Project: HBase
  Issue Type: Sub-task
  Components: Client, IPC/RPC
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 0.96.0

 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5776) HTableMultiplexer

[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539977#comment-13539977
]

Ted Yu commented on HBASE-5776:
---

Integrated to trunk.

Thanks for the patch, Liyin and lijin.

Thanks for the review, Ram.

HTableMultiplexer
--

Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Fix For: 0.96.0

Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch,
5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch

There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.

To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539979#comment-13539979
 ] 

Ted Yu commented on HBASE-7400:
---

I see.
What do you think about my other comment w.r.t. 'new Configuration()' ?

Thanks

 ExportSnapshot mapper closes the FileSystem
 ---

 Key: HBASE-7400
 URL: https://issues.apache.org/jira/browse/HBASE-7400
 Project: HBase
  Issue Type: Sub-task
  Components: Client, master, regionserver, snapshots, Zookeeper
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Blocker
 Fix For: hbase-6055, 0.96.0

 Attachments: HBASE-7400-v0.patch


 The ExportSnapshotMapper closes the FileSystem on cleanup,
 and in case of shared instead, since there's no ref count, the job fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7428) There are multiple HConstants for configuring Zookeeper timeout

2012-12-27 Thread Nick Dimiduk (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-7428:


Status: Open  (was: Patch Available)

 There are multiple HConstants for configuring Zookeeper timeout
 ---

 Key: HBASE-7428
 URL: https://issues.apache.org/jira/browse/HBASE-7428
 Project: HBase
  Issue Type: Improvement
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
  Labels: noob
 Fix For: 0.96.0

 Attachments: 7428-consolidate-ZK_SESSION_TIMEOUT.0.diff


 From [~te...@apache.org] to dev@:
 {quote}
 There are two constants with the same value:
 HConstants.ZOOKEEPER_SESSION_TIMEOUT and HConstants.ZK_SESSION_TIMEOUT
 HConstants.ZOOKEEPER_SESSION_TIMEOUT is only used in tests.
 HConstants.ZK_SESSION_TIMEOUT is used by ZKUtil
 Shall we remove HConstants.ZOOKEEPER_SESSION_TIMEOUT and let tests use
 HConstants.ZK_SESSION_TIMEOUT ?
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7428) There are multiple HConstants for configuring Zookeeper timeout

2012-12-27 Thread Nick Dimiduk (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-7428:


Status: Patch Available  (was: Open)

 There are multiple HConstants for configuring Zookeeper timeout
 ---

 Key: HBASE-7428
 URL: https://issues.apache.org/jira/browse/HBASE-7428
 Project: HBase
  Issue Type: Improvement
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
  Labels: noob
 Fix For: 0.96.0

 Attachments: 7428-consolidate-ZK_SESSION_TIMEOUT.0.diff


 From [~te...@apache.org] to dev@:
 {quote}
 There are two constants with the same value:
 HConstants.ZOOKEEPER_SESSION_TIMEOUT and HConstants.ZK_SESSION_TIMEOUT
 HConstants.ZOOKEEPER_SESSION_TIMEOUT is only used in tests.
 HConstants.ZK_SESSION_TIMEOUT is used by ZKUtil
 Shall we remove HConstants.ZOOKEEPER_SESSION_TIMEOUT and let tests use
 HConstants.ZK_SESSION_TIMEOUT ?
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file


 [ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-5598:
---

Attachment: 5598.part2.patch

 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
 Fix For: 0.96.0

 Attachments: 5598.part1.patch, 5598.part2.patch, 
 ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file

[
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1354#comment-1354
]

nkeywal commented on HBASE-5598:

part2:
- adds the dependency to findbugs for the annotations. It's a compile only
dependency. License is LGPL.
- suppress a few not critical warnings
- fixes a few others

Locally, I'm now at 144.

Analyse and fix the findbugs reporting by QA and add invalid bugs into
findbugs-excludeFilter file
--

Key: HBASE-5598
URL: https://issues.apache.org/jira/browse/HBASE-5598
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
Fix For: 0.96.0

Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch,
ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch

There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to
up the OK count.
This may lead to other issues when we re-factor the code, if we induce new
valid ones and remove invalid bugs also can not be reported by QA.
So, I would propose to add the exclude filter file for findbugs(for the
invalid bugs). If we find any valid ones, we can fix under this JIRA.

[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file


 [ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-5598:
---

Status: Open  (was: Patch Available)

 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.94.0, 0.92.1, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
 Fix For: 0.96.0

 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, 
 ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file


 [ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-5598:
---

Attachment: 5598.part2.patch

 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
 Fix For: 0.96.0

 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, 
 ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7442) HBase remote CopyTable not working when security enabled

James Kinley created HBASE-7442:
---

 Summary: HBase remote CopyTable not working when security enabled
 Key: HBASE-7442
 URL: https://issues.apache.org/jira/browse/HBASE-7442
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, mapreduce, security
Affects Versions: 0.92.1
Reporter: James Kinley


When security is enabled, HBase CopyTable fails with Kerberos exception:

{code}
FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most 
likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
{code}

This is only when copying to remote HBase cluster (using either MRv1 or YARN), 
local copy works fine.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file


 [ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-5598:
---

Status: Patch Available  (was: Open)

 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.94.0, 0.92.1, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
 Fix For: 0.96.0

 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, 
 ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6752) On region server failure, serve writes and timeranged reads during the log split


[ 
https://issues.apache.org/jira/browse/HBASE-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540005#comment-13540005
 ] 

nkeywal commented on HBASE-6752:


@Gregory: As you have unassigned the jira, I will have a look in the next 
weeks. Have you studied some options in more details and rejected them?

 On region server failure, serve writes and timeranged reads during the log 
 split
 

 Key: HBASE-6752
 URL: https://issues.apache.org/jira/browse/HBASE-6752
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Priority: Minor

 Opening for write on failure would mean:
 - Assign the region to a new regionserver. It marks the region as recovering
   -- specific exception returned to the client when we cannot server.
   -- allow them to know where they stand. The exception can include some time 
 information (failure stated on: ...)
   -- allow them to go immediately on the right regionserver, instead of 
 retrying or calling the region holding meta to get the new address
  = save network calls, lower the load on meta.
 - Do the split as today. Priority is given to region server holding the new 
 regions
   -- help to share the load balancing code: the split is done by region 
 server considered as available for new regions
   -- help locality (the recovered edits are available on the region server) 
 = lower the network usage
 - When the split is finished, we're done as of today
 - while the split is progressing, the region server can
  -- serve writes
--- that's useful for all application that need to write but not read 
 immediately:
--- whatever logs events to analyze them later
--- opentsdb is a perfect example.   
  -- serve reads if they have a compatible time range. For heavily used 
 tables, it could be an help, because:
--- we can expect to have a few minutes of data only (as it's loaded)
--- the heaviest queries, often accepts a few -or more- minutes delay. 
 Some What if:
 1) the split fails
 = Retry until it works. As today. Just that we serves writes. We need to 
 know (as today) that the region has not recovered if we fail again.
 2) the regionserver fails during the split
 = As 1 and as of today/
 3) the regionserver fails after the split but before the state change to 
 fully available.
 = New assign. More logs to split (the ones already dones and the new ones).
 4) the assignment fails
 = Retry until it works. As today.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled


[ 
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540007#comment-13540007
 ] 

James Kinley commented on HBASE-7442:
-

To give some context for the attached log 
({{attempt_201212271546_0001_m_00_0.log}}).

I have setup 2 clusters:

Cluster 1 has two nodes running the following services:
one.kinley.com: HDFS, MRv1
two.kinley.com: HBase, ZK

Cluster 2 has a single node running the following:
three.kinley.com: HDFS, HBase, ZK

Security is enabled on both clusters and all principals are in the same realm 
(so cross-realm support is not required).

I'm trying to copy an HBase table from cluster 1 to cluster 2:

{code}
two.kinley.com:~$ hbase org.apache.hadoop.hbase.mapreduce.CopyTable 
-Dmapred.child.java.opts=-Dsun.security.krb5.debug=true 
-Dmapred.map.child.log.level=DEBUG --peer.adr=three.kinley.com:2181:/hbase 
--new.name=Converged-headers Converged-headers
{code}

Running a local2local CopyTable on cluster 1 and 2 works ok.

 HBase remote CopyTable not working when security enabled
 

 Key: HBASE-7442
 URL: https://issues.apache.org/jira/browse/HBASE-7442
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, mapreduce, security
Affects Versions: 0.92.1
Reporter: James Kinley

 When security is enabled, HBase CopyTable fails with Kerberos exception:
 {code}
 FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The 
 most likely cause is missing or invalid credentials. Consider 'kinit'.
 javax.security.sasl.SaslException: GSS initiate failed [Caused by 
 GSSException: No valid credentials provided (Mechanism level: Failed to find 
 any Kerberos tgt)]
 {code}
 This is only when copying to remote HBase cluster (using either MRv1 or 
 YARN), local copy works fine.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7442) HBase remote CopyTable not working when security enabled


 [ 
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Kinley updated HBASE-7442:


Attachment: attempt_201212271546_0001_m_00_0.log

 HBase remote CopyTable not working when security enabled
 

 Key: HBASE-7442
 URL: https://issues.apache.org/jira/browse/HBASE-7442
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, mapreduce, security
Affects Versions: 0.92.1
Reporter: James Kinley
 Attachments: attempt_201212271546_0001_m_00_0.log


 When security is enabled, HBase CopyTable fails with Kerberos exception:
 {code}
 FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The 
 most likely cause is missing or invalid credentials. Consider 'kinit'.
 javax.security.sasl.SaslException: GSS initiate failed [Caused by 
 GSSException: No valid credentials provided (Mechanism level: Failed to find 
 any Kerberos tgt)]
 {code}
 This is only when copying to remote HBase cluster (using either MRv1 or 
 YARN), local copy works fine.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file


[ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540027#comment-13540027
 ] 

stack commented on HBASE-5598:
--

@nkeywal Do you want to do part2 in another issue linked to from here since we 
have already committed a patch against this issue?

Why move the defines in HRegionInfo?

This is a nice trick:

{code}
+@edu.umd.cs.findbugs.annotations.SuppressWarnings 
(value=ES_COMPARING_STRINGS_WITH_EQ,
+justification=Optimization)
{code}

What warning do we suppress by doing this:

-  return new StringBuilder(AuthResult)
-  .append(toContextString()).toString();
+  return AuthResult + toContextString();


I know its not 'important' but these kinda findings are good:

-  StringBuffer sb = new StringBuffer();
+  StringBuilder sb = new StringBuilder();

We are going back to findbugs 2.0.1 from 2.5.2?

Is this define used anywhere: findbugs-maven-plugin.version

Good work Nicolas


 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
 Fix For: 0.96.0

 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, 
 ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled


[ 
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540031#comment-13540031
 ] 

James Kinley commented on HBASE-7442:
-

I noticed that *{{CopyTable#createSubmittableJob}}*, and specifically 
*{{TableMapReduceUtil#initCredentials}}* was only acquiring a delegation token 
for the local HBase cluster so I added the following to 
*{{TableMapReduceUtil#initCredentials}}*:

{code}
String quorumAddress = 
job.getConfiguration().get(TableOutputFormat.QUORUM_ADDRESS);
if (quorumAddress != null) {
  String[] parts = ZKUtil.transformClusterKey(quorumAddress);  
  Configuration peerConf = HBaseConfiguration.create(job.getConfiguration());
  peerConf.set(HConstants.ZOOKEEPER_QUORUM, parts[0]);
  peerConf.set(hbase.zookeeper.client.port, parts[1]);
  peerConf.set(HConstants.ZOOKEEPER_ZNODE_PARENT, parts[2]);
  User.getCurrent().obtainAuthTokenForJob(peerConf, job);
}
{code}

The user now has the correct tokens:

Cluster 1 ID = 7bb01c84-3c66-49ca-82d2-2f9218f13dad
Cluster 2 ID = d4653a37-83df-4929-b950-e03987c3b4e8

{code}
INFO token.TokenUtil: Obtained token HBASE_AUTH_TOKEN for user 
kinley/ad...@kinley.com on cluster 7bb01c84-3c66-49ca-82d2-2f9218f13dad
INFO token.TokenUtil: Obtained token HBASE_AUTH_TOKEN for user 
kinley/ad...@kinley.com on cluster d4653a37-83df-4929-b950-e03987c3b4e8
{code}

But when initialising the *{{HTable}}* connection in the mapper, the 
*{{TokenSelector}}* in *{{SecureClient#SecureConnection}}* returns the wrong 
token from the user's token cache. This is because the token cache is keyed on 
{{clusterId}}, and despite the token for the local cluster existing in the 
cache ({{key=7bb01c84-3c66-49ca-82d2-2f9218f13dad}}), the {{clusterId}} is 
fixed at this point in *{{HBaseClient}}* to the remote cluster's ID 
({{key=d4653a37-83df-4929-b950-e03987c3b4e8}}), and it is this ID that is 
always passed to the *{{TokenSelector}}*.

I can think of a couple of possible solutions:
1. Either pass the {{clusterId}} into *{{SecureClient#SecureConnection}}* as 
part of the *{{ConnectionId}}*. This way it can be used by the 
*{{TokenSelector}}* instead of the fixed ID in the parent class 
*{{HBaseClient}}*?, or 
2. *{{SecureRpcEngine}}* caches multiple clients based on a 
*{{SocketFactory}}*, so we could maintain separate clients for both the local 
and remote clusters here?

 HBase remote CopyTable not working when security enabled
 

 Key: HBASE-7442
 URL: https://issues.apache.org/jira/browse/HBASE-7442
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, mapreduce, security
Affects Versions: 0.92.1
Reporter: James Kinley
 Attachments: attempt_201212271546_0001_m_00_0.log


 When security is enabled, HBase CopyTable fails with Kerberos exception:
 {code}
 FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The 
 most likely cause is missing or invalid credentials. Consider 'kinit'.
 javax.security.sasl.SaslException: GSS initiate failed [Caused by 
 GSSException: No valid credentials provided (Mechanism level: Failed to find 
 any Kerberos tgt)]
 {code}
 This is only when copying to remote HBase cluster (using either MRv1 or 
 YARN), local copy works fine.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7400) ExportSnapshot mapper closes the FileSystem


 [ 
https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-7400:
---

Attachment: HBASE-7400-v1.patch

sorry Ted, I tend to forget to reply, to comments/nits/suggestions that I agree 
with, and I'm going change the code.

Anyway, you're right.
The configuration doesn't matter since the user should specify a qualified path 
hdfs://cluster2
and in case of unqualified path, you've an unexpected result in both case. 
but at least using the getConf()
you're sure that you get all your config (HBaseConfig is used) and maybe your 
fs.default.name is inside one of those.

 ExportSnapshot mapper closes the FileSystem
 ---

 Key: HBASE-7400
 URL: https://issues.apache.org/jira/browse/HBASE-7400
 Project: HBase
  Issue Type: Sub-task
  Components: Client, master, regionserver, snapshots, Zookeeper
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Blocker
 Fix For: hbase-6055, 0.96.0

 Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch


 The ExportSnapshotMapper closes the FileSystem on cleanup,
 and in case of shared instead, since there's no ref count, the job fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540039#comment-13540039
 ] 

Ted Yu commented on HBASE-7400:
---

Thanks for the quick response.
+1 on second patch.

 ExportSnapshot mapper closes the FileSystem
 ---

 Key: HBASE-7400
 URL: https://issues.apache.org/jira/browse/HBASE-7400
 Project: HBase
  Issue Type: Sub-task
  Components: Client, master, regionserver, snapshots, Zookeeper
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Blocker
 Fix For: hbase-6055, 0.96.0

 Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch


 The ExportSnapshotMapper closes the FileSystem on cleanup,
 and in case of shared instead, since there's no ref count, the job fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5776) HTableMultiplexer


[ 
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540047#comment-13540047
 ] 

Hudson commented on HBASE-5776:
---

Integrated in HBase-TRUNK #3660 (See 
[https://builds.apache.org/job/HBase-TRUNK/3660/])
HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java


 HTableMultiplexer 
 --

 Key: HBASE-5776
 URL: https://issues.apache.org/jira/browse/HBASE-5776
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
 Fix For: 0.96.0

 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 
 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.1.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.1.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.2.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.2.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.3.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.4.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, 
 HBASE-5776-trunk-V2.patch


 There is a known issue in HBase client that single slow/dead region server 
 could slow down the multiput operations across all the region servers. So the 
 HBase client will be as slow as the slowest region server in the cluster. 
  
 To solve this problem, HTableMultiplexer will separate the multiput 
 submitting threads with the flush threads, which means the multiput operation 
 will be a nonblocking operation. 
 The submitting thread will shard all the puts into different queues based on 
 its destination region server and return immediately. The flush threads will 
 flush these puts from each queue to its destination region server. 
 Currently the HTableMultiplexer only supports the put operation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7442) HBase remote CopyTable not working when security enabled


 [ 
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-7442:
-

Fix Version/s: 0.94.5
   0.96.0

 HBase remote CopyTable not working when security enabled
 

 Key: HBASE-7442
 URL: https://issues.apache.org/jira/browse/HBASE-7442
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, mapreduce, security
Affects Versions: 0.92.1
Reporter: James Kinley
 Fix For: 0.96.0, 0.94.5

 Attachments: attempt_201212271546_0001_m_00_0.log


 When security is enabled, HBase CopyTable fails with Kerberos exception:
 {code}
 FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The 
 most likely cause is missing or invalid credentials. Consider 'kinit'.
 javax.security.sasl.SaslException: GSS initiate failed [Caused by 
 GSSException: No valid credentials provided (Mechanism level: Failed to find 
 any Kerberos tgt)]
 {code}
 This is only when copying to remote HBase cluster (using either MRv1 or 
 YARN), local copy works fine.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled


[ 
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540051#comment-13540051
 ] 

Lars Hofhansl commented on HBASE-7442:
--

Thanks James.

[~apurtell], [~ghelmling] any thoughts?

 HBase remote CopyTable not working when security enabled
 

 Key: HBASE-7442
 URL: https://issues.apache.org/jira/browse/HBASE-7442
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, mapreduce, security
Affects Versions: 0.92.1
Reporter: James Kinley
 Fix For: 0.96.0, 0.94.5

 Attachments: attempt_201212271546_0001_m_00_0.log


 When security is enabled, HBase CopyTable fails with Kerberos exception:
 {code}
 FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The 
 most likely cause is missing or invalid credentials. Consider 'kinit'.
 javax.security.sasl.SaslException: GSS initiate failed [Caused by 
 GSSException: No valid credentials provided (Mechanism level: Failed to find 
 any Kerberos tgt)]
 {code}
 This is only when copying to remote HBase cluster (using either MRv1 or 
 YARN), local copy works fine.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file


[ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540050#comment-13540050
 ] 

nkeywal commented on HBASE-5598:


bq. @nkeywal Do you want to do part2 in another issue linked to from here since 
we have already committed a patch against this issue?
Ok, I will do that.

bq. Why move the defines in HRegionInfo?
We're constructing an object while the static fields are not yet all 
initialized:
{code}
class A{
 static A = new A();
 static int i = 15;
 A(){ print i= + i; }
}
{code}

prints i=0;

Tricky isn't it?

bq. return new StringBuilder(AuthResult)
.append(toContextString()).toString();
+ return AuthResult + toContextString();

Don't remember its name (it could be an IntelliJ warning as well, I fix both), 
but it's just that the new code is simpler and at least as efficient as the 
previous one, as AuthResult is a constant.

bq. We are going back to findbugs 2.0.1 from 2.5.2?
There are two versions, one for findbugs, one for the maven plugin. Maven 
plugin 2.5.2 includes findbugs 2.0.1.
The naming was previously confusing, so I renamed the variables.
And we need to explicitly define the two versions, as we're including the 
plugin but as well the annotations. 
Last findbugs is 2.0.2, but I put 2.0.1 to match the version included by the 
maven plugin

bq. Is this define used anywhere: findbugs-maven-plugin.version
Yep, in the main pom, when including the maven plugin.



 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
 Fix For: 0.96.0

 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, 
 ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7443) More findbugs fixes

nkeywal created HBASE-7443:
--

 Summary: More findbugs fixes
 Key: HBASE-7443
 URL: https://issues.apache.org/jira/browse/HBASE-7443
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.96.0
 Attachments: 7443.patch

It   
- adds the dependency to findbugs for the annotations. It's a compile only 
dependency. License is LGPL.
- suppresses a few not critical warnings
- fixes a few others

Locally, I'm now at 144.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.

[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540054#comment-13540054
]

Ted Yu commented on HBASE-5416:
---

bq. Now in this case i will have only the joinedScanner heap.
I think you meant we only have storeHeap.

Here is related code from HRegion.nextInternal() of 0.94 branch:
{code}
if (isStopRow(currentRow, offset, length)) {
if (filter != null filter.hasFilterRow()) {
filter.filterRow(results);
}
if (filter != null filter.filterRow()) {
results.clear();
}

return false;
{code}
We can see the logic is the same as that in patch v12.

Improve performance of scans with some kind of filters.
---

Key: HBASE-5416
URL: https://issues.apache.org/jira/browse/HBASE-5416
Project: HBase
Issue Type: Improvement
Components: Filters, Performance, regionserver
Affects Versions: 0.90.4
Reporter: Max Lapan
Assignee: Sergey Shelukhin
Fix For: 0.96.0

Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt,
Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch,
Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch,
Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch,
HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch,
HBASE-5416-v8.patch, HBASE-5416-v9.patch

When the scan is performed, whole row is loaded into result list, after that
filter (if exists) is applied to detect that row is needed.
But when scan is performed on several CFs and filter checks only data from
the subset of these CFs, data from CFs, not checked by a filter is not needed
on a filter stage. Only when we decided to include current row. And in such
case we can significantly reduce amount of IO performed by a scan, by loading
only values, actually checked by a filter.
For example, we have two CFs: flags and snap. Flags is quite small (bunch of
megabytes) and is used to filter large entries from snap. Snap is very large
(10s of GB) and it is quite costly to scan it. If we needed only rows with
some flag specified, we use SingleColumnValueFilter to limit result to only
small subset of region. But current implementation is loading both CFs to
perform scan, when only small subset is needed.
Attached patch adds one routine to Filter interface to allow filter to
specify which CF is needed to it's operation. In HRegion, we separate all
scanners into two groups: needed for filter and the rest (joined). When new
row is considered, only needed data is loaded, filter applied, and only if
filter accepts the row, rest of data is loaded. At our data, this speeds up
such kind of scans 30-50 times. Also, this gives us the way to better
normalize the data into separate columns by optimizing the scans performed.

[jira] [Updated] (HBASE-7443) More findbugs fixes


 [ 
https://issues.apache.org/jira/browse/HBASE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-7443:
---

Attachment: 7443.patch

 More findbugs fixes
 ---

 Key: HBASE-7443
 URL: https://issues.apache.org/jira/browse/HBASE-7443
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.96.0

 Attachments: 7443.patch


 It   
 - adds the dependency to findbugs for the annotations. It's a compile only 
 dependency. License is LGPL.
 - suppresses a few not critical warnings
 - fixes a few others
 Locally, I'm now at 144.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5416) Improve performance of scans with some kind of filters.

[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ted Yu updated HBASE-5416:
--

Attachment: 5416-v13.patch

Patch v13 rebases on trunk.
TestSingleColumnValueExcludeFilter, TestHRegion and TestJoinedScanners pass.

Improve performance of scans with some kind of filters.
---

Attachments: 5416-Filtered_scans_v6.patch, 5416-v13.patch,
5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch,
Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch,
Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch,
HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch,
HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch

[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.

2012-12-27 Thread Max Lapan (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540062#comment-13540062
]

Max Lapan commented on HBASE-5416:
--

bq. I think you meant we only have storeHeap.

No, exactly one KVS in joinedScanner heap and empty storeHeap. It was caused by
{{!scan.doLoadColumnFamiliesOnDemand()}} extra condition in constructor.

Improve performance of scans with some kind of filters.
---

[jira] [Commented] (HBASE-7390) Add extra test cases for assignement on the region server and fix the related issues

2012-12-27 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540064#comment-13540064
 ] 

Jimmy Xiang commented on HBASE-7390:


Here is the link to the review:  https://reviews.apache.org/r/8754/

I will take a look today.

 Add extra test cases for assignement on the region server and fix the related 
 issues
 

 Key: HBASE-7390
 URL: https://issues.apache.org/jira/browse/HBASE-7390
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment, regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
 Fix For: 0.96.0

 Attachments: 7390.v1.patch, 7390.v2.patch, 7390.v3.patch, 
 7390.v4.patch, 7390.v6.patch, 7390.v7.patch, 7390.v8.patch, 
 assignment_zk_states.jpg


 We don't have a lot of tests on the region server itself.
 Here are some.
 Some of them are failing, feedback welcome.
 See as well the attached state diagram for the ZK nodes on assignment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7258) Hbase needs to create baseZNode recursivly

2012-12-27 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540068#comment-13540068
 ] 

ramkrishna.s.vasudevan commented on HBASE-7258:
---

@Stack/Ted
I have given my word to Liu on commiting this.  Pls let me know if we can 
commit this.  

 Hbase needs to create baseZNode recursivly
 --

 Key: HBASE-7258
 URL: https://issues.apache.org/jira/browse/HBASE-7258
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.2
Reporter: Liu Shaohui
Priority: Minor
  Labels: zookeeper, zookeeper.znode.parent
 Attachments: HBASE-7258-0.94.patch, HBASE-7258.diff, HBASE-7258.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 In deploy env, multi small hbase clusters may share a same zk cluster. So, 
 for hbase cluster1, its parent znode is /hbase/cluster1. But in hbase version 
 0.94.1, hbase use ZKUtil.createAndFailSilent(this, baseZNode) to create 
 parent path and it will throw a NoNode exception if znode /hbase donot exist.
 We want to change it to ZKUtil.createWithParents(this, baseZNode); to suport 
 create baseZNode recursivly. 
 The NoNode exception is:
 java.lang.RuntimeException: Failed construction of Master: class 
 org.apache.hadoop.hbase.master.HMaster
 at 
 org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1792)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at 
 org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:77)
 at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806)
 Caused by: org.apache.zookeeper.KeeperException$NoNodeException: 
 KeeperErrorCode = NoNode for /hbase/cluster1
 at 
 org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
 at 
 org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:420)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:402)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:905)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:166)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:159)
 at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:282)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at org.apache.hadoop.hbase.master.H

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.

2012-12-27 Thread ramkrishna.s.vasudevan (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540069#comment-13540069
]

ramkrishna.s.vasudevan commented on HBASE-5416:
---

Yes Max that's what i meant.
Ted, just take the condition where isFamilyEssential = true. Also scan says
lazy loading of CF is allowed. (and also only one CF).
So storeHeap will be null in this case.

Improve performance of scans with some kind of filters.
---

[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.

[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540070#comment-13540070
]

Ted Yu commented on HBASE-5416:
---

Here is the change related to Max's comment above:
{code}
+if (this.filter == null || !scan.doLoadColumnFamiliesOnDemand()
+ || this.filter.isFamilyEssential(entry.getKey())) {
+ scanners.add(scanner);
{code}
Ram said:
bq. Also scan says lazy loading of CF is allowed.
So the condition {{!scan.doLoadColumnFamiliesOnDemand()}} should be false,
right ?

Improve performance of scans with some kind of filters.
---

[jira] [Commented] (HBASE-6752) On region server failure, serve writes and timeranged reads during the log split

2012-12-27 Thread Gregory Chanan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540076#comment-13540076
 ] 

Gregory Chanan commented on HBASE-6752:
---

@nkeywal: didn't study anything in too much depth.

For the read part, my thought was to implement a config (in HTableDescriptor?) 
that would reject user-set timestamps on writes, so we know for sure there 
can't be any writes in the timestamp range that need to be replayed from the 
WAL.  I suspect there are other optimizations we could do with that 
information, but haven't thought it through.

For writes, do you create a new WAL for the new writes that are happening while 
the log is still replaying?  If so, management could be complicated and it 
might make sense to have support for multiple WALs already before tackling 
that.  If not (you write to the same WAL), would that even work?  I guess you 
would want to avoid replaying the new writes (might be okay if all WAL updates 
are idempotent, but could be an issue if a lot of writes go in during the 
replay time).

 On region server failure, serve writes and timeranged reads during the log 
 split
 

 Key: HBASE-6752
 URL: https://issues.apache.org/jira/browse/HBASE-6752
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Priority: Minor

 Opening for write on failure would mean:
 - Assign the region to a new regionserver. It marks the region as recovering
   -- specific exception returned to the client when we cannot server.
   -- allow them to know where they stand. The exception can include some time 
 information (failure stated on: ...)
   -- allow them to go immediately on the right regionserver, instead of 
 retrying or calling the region holding meta to get the new address
  = save network calls, lower the load on meta.
 - Do the split as today. Priority is given to region server holding the new 
 regions
   -- help to share the load balancing code: the split is done by region 
 server considered as available for new regions
   -- help locality (the recovered edits are available on the region server) 
 = lower the network usage
 - When the split is finished, we're done as of today
 - while the split is progressing, the region server can
  -- serve writes
--- that's useful for all application that need to write but not read 
 immediately:
--- whatever logs events to analyze them later
--- opentsdb is a perfect example.   
  -- serve reads if they have a compatible time range. For heavily used 
 tables, it could be an help, because:
--- we can expect to have a few minutes of data only (as it's loaded)
--- the heaviest queries, often accepts a few -or more- minutes delay. 
 Some What if:
 1) the split fails
 = Retry until it works. As today. Just that we serves writes. We need to 
 know (as today) that the region has not recovered if we fail again.
 2) the regionserver fails during the split
 = As 1 and as of today/
 3) the regionserver fails after the split but before the state change to 
 fully available.
 = New assign. More logs to split (the ones already dones and the new ones).
 4) the assignment fails
 = Retry until it works. As today.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7440) ReplicationZookeeper#addPeer is racy

2012-12-27 Thread Himanshu Vashishtha (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Himanshu Vashishtha updated HBASE-7440:
---

Attachment: HBASE-7440-v2.patch

Using ZKUtil.createNodeIfNotExistsAndWatch as suggested by Lars. Didn't notice 
this API earlier!
Yes Matteo, the default value of znode is ENABLED, and this code is called 
while creating one. So, no check needed or such.

 ReplicationZookeeper#addPeer is racy
 

 Key: HBASE-7440
 URL: https://issues.apache.org/jira/browse/HBASE-7440
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.3
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
 Fix For: 0.96.0, 0.94.4

 Attachments: HBASE-7440-v0.patch, HBASE-7440-v1.patch, 
 HBASE-7440-v2.patch


 While adding a peer, ReplicationZK does the znodes creation in three 
 transactions. Create :
 a) peers znode
 b) peerId specific znode, and
 c) peerState znode
 There is a PeerWatcher which invokes getPeer() (after steps b) and c)). If it 
 happens that while adding a peer, the control flows to getPeer() and step c) 
 has not been processed, it may results in a state where the peer will not be 
 added. This happens while running 
 TestMasterReplication#testCyclicReplication().
 {code}
 2012-12-26 07:36:35,187 INFO  
 [RegionServer:0;p0120.X,38423,1356536179470-EventThread] 
 zookeeper.RecoverableZooKeeper(447): Node /2/replication/peers/1/peer-state 
 already exists and this is not a retry
 2012-12-26 07:36:35,188 ERROR 
 [RegionServer:0;p0120.X,38423,1356536179470-EventThread] 
 regionserver.ReplicationSourceManager$PeersWatcher(527): Error while adding a 
 new peer
 org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
 NodeExists for /2/replication/peers/1/peer-state
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:428)
   at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:410)
   at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:1044)
   at 
 org.apache.hadoop.hbase.replication.ReplicationPeer.startStateTracker(ReplicationPeer.java:82)
   at 
 org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:344)
   at 
 org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:307)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:519)
   at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:315)
   at 
 org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
 2012-12-26 07:36:35,188 DEBUG 
 [RegionServer:0;p0120.X,55742,1356536171947-EventThread] 
 zookeeper.ZKUtil(1545): regionserver:55742-0x13bd7db39580004 Retrieved 36 
 byte(s) of data from znode /1/hbaseid; data=9ce66123-d3e8-4ae9-a249-afe03...
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.

[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540078#comment-13540078
]

Ted Yu commented on HBASE-5416:
---

Here is code from patch v7:
{code}
+if (this.filter == null ||
this.filter.isFamilyEssential(entry.getKey())) {
+ scanners.add(scanner);
{code}
In the above case, I don't see difference between v7 and v12 w.r.t. populating
scanners List.

Improve performance of scans with some kind of filters.
---

[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment

2012-12-27 Thread Gregory Chanan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540080#comment-13540080
 ] 

Gregory Chanan commented on HBASE-7263:
---

bq. Does your test just operate one rowkey?

The full test description is given above.  The relevant part to your question:
- Each increment uniformly distributed over 500,000 rows

bq.  Do you mean it is not per-row for option 2( Have an MVCC per-row (table 
configuration): this avoids the unnecessary contention of 1))
Option 2 is per-row but that is not what is implemented today and this JIRA 
takes a different approach.

 Investigate more fine grained locking for checkAndPut/append/increment
 --

 Key: HBASE-7263
 URL: https://issues.apache.org/jira/browse/HBASE-7263
 Project: HBase
  Issue Type: Improvement
  Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor

 HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
 {quote}
 1) Waiting for the MVCC to advance for read/updates: the downside is that you 
 have to wait for updates on other rows.
 2) Have an MVCC per-row (table configuration): this avoids the unnecessary 
 contention of 1)
 3) Transform the read/updates to write-only with rollup on read.. E.g. an 
 increment would just have the number of values to increment.
 {quote}
 HBASE-7051 and HBASE-4583 implement option #1.  The downside, as mentioned, 
 is that you have to wait for updates on other rows, since MVCC is per-row.
 Another option occurred to me that I think is worth investigating: rely on a 
 row-level read/write lock rather than MVCC.
 Here is pseudo-code for what exists today for read/updates like checkAndPut
 {code}
 (1)  Acquire RowLock
 (1a) BeginMVCC + Finish MVCC
 (2)  Begin MVCC
 (3)  Do work
 (4)  Release RowLock
 (5)  Append to WAL
 (6)  Finish MVCC
 {code}
 Write-only operations (e.g. puts) are the same, just without step 1a.
 Now, consider the following instead:
 {code}
 (1)  Acquire RowLock
 (1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
 (1b) Grab RowReadLock (new step!)
 (2)  Begin MVCC
 (3)  Do work
 (4)  Release RowLock
 (5)  Append to WAL
 (6)  Finish MVCC
 (7)  Release RowReadLock (new step!)
 {code}
 As before, write-only operations are the same, just without step 1a.
 The difference here is that writes grab a row-level read lock and hold it 
 until the MVCC is completed.  The nice property that this gives you is that 
 read/updates can tell when the MVCC is done on a per-row basis, because they 
 can just try to acquire the write-lock which will block until the MVCC is 
 competed for that row in step 7.
 There is overhead for acquiring the read lock that I need to measure, but it 
 should be small, since there will never be any blocking on acquiring the 
 row-level read lock.  This is because the read lock can only block if someone 
 else holds the write lock, but both the write and read lock are only acquired 
 under the row lock.
 I ran a quick test of this approach over a region (this directly interacts 
 with HRegion, so no client effects):
 - 30 threads
 - 5000 increments per thread
 - 30 columns per increment
 - Each increment uniformly distributed over 500,000 rows
 - 5 trials
 Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
 Today: 13950 ms
 The locking approach: 10877 ms
 So it looks like an improvement, at least wrt increment.  As mentioned, I 
 need to measure the overhead of acquiring the read lock for puts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.

2012-12-27 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540105#comment-13540105
]

Hadoop QA commented on HBASE-5416:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562493/5416-v13.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.

{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.

{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100

{color:red}-1 core tests{color}. The patch failed these unit tests:

org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.replication.TestReplication

{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3718//console

This message is automatically generated.

Improve performance of scans with some kind of filters.
---

[jira] [Created] (HBASE-7444) [89-fb] Update the default user name in MiniHBaseCluster

2012-12-27 Thread Liyin Tang (JIRA)

Liyin Tang created HBASE-7444:
-

 Summary: [89-fb] Update the default user name in MiniHBaseCluster
 Key: HBASE-7444
 URL: https://issues.apache.org/jira/browse/HBASE-7444
 Project: HBase
  Issue Type: Test
Reporter: Liyin Tang
Priority: Minor


Currently we are using $username.hrs.$index as default user name in 
MiniHBaseCluster, which actually is not a legal user name. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5945) Reduce buffer copies in IPC server response path

2012-12-27 Thread Devaraj Das (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Devaraj Das updated HBASE-5945:
---

Attachment: 5945-in-progress.patch

[~stack], hope it is okay with you that I am attaching a patch here... (I was
tempted to do a patch for this issue..)

The main change in this patch is that it removes the RpcRequestBody from
RPC.proto, and instead serializes the rpc method argument directly to the
underlying output. There is a lot of change associated with this removal in the
patch.

I didn't change the serialization of the rpc header fields yet (as was done by
[~tlipcon] in his earlier patch). The reason being that I don't think the
header part is a concern since they will be a few 10s of bytes and mostly will
be noise. The ser/de of the rpc request-body/response-body is where I tried to
improve on. The codebase compiles with this patch now but there is work still
left (and maybe I missed some buffer copies as well; need to dig some more).

Reduce buffer copies in IPC server response path

Key: HBASE-5945
URL: https://issues.apache.org/jira/browse/HBASE-5945
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Affects Versions: 0.96.0
Reporter: Todd Lipcon
Assignee: stack
Priority: Blocker
Fix For: 0.96.0

Attachments: 5945-in-progress.patch, buffer-copies.txt,
even-fewer-copies.txt, hbase-5495.txt

The new PB code is sloppy with buffers and makes several needless copies.
This increases GC time a lot. A few simple changes can cut this back down.

[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine


[ 
https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540138#comment-13540138
 ] 

stack commented on HBASE-5705:
--

It looks like this patch made it so that when we set up the proxy, we no longer 
ask the server what its version of the protocol is.  Instead we return the 
client's version of the protocol.  See below.

+  static long getProtocolVersion(Class? extends VersionedProtocol protocol)
+  throws NoSuchFieldException, IllegalAccessException {
+Field versionField = protocol.getField(VERSION);
+versionField.setAccessible(true);
+return versionField.getLong(protocol);
+  }

Was expediency the reason the querying of server version was undone?  Something 
to be addressed later when VersionedProtocol was fixed up?

Thanks.

 Introduce Protocol Buffer RPC engine
 

 Key: HBASE-5705
 URL: https://issues.apache.org/jira/browse/HBASE-5705
 Project: HBase
  Issue Type: Sub-task
  Components: IPC/RPC, master, migration, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
 Fix For: 0.96.0

 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch, 
 5705-2.3.patch


 Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB 
 aware can be made to go through this RPC engine. The approach, in my current 
 thinking, would be similar to HADOOP-7773.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7188) Move classes into hbase-client


[ 
https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540139#comment-13540139
 ] 

stack commented on HBASE-7188:
--

Thanks [~eclark] Looks like something we need to reinstitute.  Will do over in 
hbase-6521.

 Move classes into hbase-client
 --

 Key: HBASE-7188
 URL: https://issues.apache.org/jira/browse/HBASE-7188
 Project: HBase
  Issue Type: Sub-task
  Components: Client, IPC/RPC
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 0.96.0

 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7188) Move classes into hbase-client


[ 
https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540145#comment-13540145
 ] 

stack commented on HBASE-7188:
--

It didn't apply clean for me.


patching file 
hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java
Reversed (or previously applied) patch detected!  Assume -R? [n] 

Am I seeing that properly, that there are packages in hbase-common now for 
regionserver and security?  Doesn't seem right.

Should hbase-default.xml be in hbase-common?

Or that version stuff




 Move classes into hbase-client
 --

 Key: HBASE-7188
 URL: https://issues.apache.org/jira/browse/HBASE-7188
 Project: HBase
  Issue Type: Sub-task
  Components: Client, IPC/RPC
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 0.96.0

 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)

2012-12-27 Thread Himanshu Vashishtha (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540146#comment-13540146
 ] 

Himanshu Vashishtha commented on HBASE-7055:


I think it would be nice to have some benchmarks for this feature against the 
existing Compaction algo?

 port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice 
 (not configurable by cf or dynamically)
 -

 Key: HBASE-7055
 URL: https://issues.apache.org/jira/browse/HBASE-7055
 Project: HBase
  Issue Type: Task
  Components: Compaction
Affects Versions: 0.96.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.96.0

 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, 
 HBASE-6371-v3-refactor-only-squashed.patch, 
 HBASE-6371-v4-refactor-only-squashed.patch, 
 HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, 
 HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch, 
 HBASE-7055-v4.patch


 There's divergence in the code :(
 See HBASE-6371 for details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled

[
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540152#comment-13540152
]

Andrew Purtell commented on HBASE-7442:
---

Thanks for identifying the problem. We should fix this (incorrect) assumption
made in the client code. I don't have a strong opinion but I like option #1
better: for the sake of simplicity a connection to a regionserver is handled
(and cached) the same way whether it is the local cluster or remote.

HBase remote CopyTable not working when security enabled

Key: HBASE-7442
URL: https://issues.apache.org/jira/browse/HBASE-7442
Project: HBase
Issue Type: Bug
Components: IPC/RPC, mapreduce, security
Affects Versions: 0.92.1
Reporter: James Kinley
Fix For: 0.96.0, 0.94.5

Attachments: attempt_201212271546_0001_m_00_0.log

When security is enabled, HBase CopyTable fails with Kerberos exception:
{code}
FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The
most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
{code}
This is only when copying to remote HBase cluster (using either MRv1 or
YARN), local copy works fine.

[jira] [Comment Edited] (HBASE-7442) HBase remote CopyTable not working when security enabled

[
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540152#comment-13540152
]

Andrew Purtell edited comment on HBASE-7442 at 12/27/12 9:08 PM:
-

Thanks for identifying the problem. We should fix this (incorrect) assumption
made in the client code. I don't have a strong opinion but I like option #1
better: for the sake of simplicity a connection is handled (and cached) the
same way whether it is to a host in the local cluster or a remote one.

was (Author: apurtell):
Thanks for identifying the problem. We should fix this (incorrect)
assumption made in the client code. I don't have a strong opinion but I like
option #1 better: for the sake of simplicity a connection to a regionserver is
handled (and cached) the same way whether it is the local cluster or remote.

HBase remote CopyTable not working when security enabled

Attachments: attempt_201212271546_0001_m_00_0.log

[jira] [Commented] (HBASE-7443) More findbugs fixes


[ 
https://issues.apache.org/jira/browse/HBASE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540161#comment-13540161
 ] 

stack commented on HBASE-7443:
--

+1 on fix.  Hoist note on new annotation up into the release notes I'd say on 
commit.  Yes, the initialize while class is being constructed is a tricky 
one findbugs finding real issues.

 More findbugs fixes
 ---

 Key: HBASE-7443
 URL: https://issues.apache.org/jira/browse/HBASE-7443
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.96.0

 Attachments: 7443.patch


 It   
 - adds the dependency to findbugs for the annotations. It's a compile only 
 dependency. License is LGPL.
 - suppresses a few not critical warnings
 - fixes a few others
 Locally, I'm now at 144.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)

2012-12-27 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540176#comment-13540176
 ] 

Jimmy Xiang commented on HBASE-7055:


I like the idea of this feature. I think it is useful. However, I am not sure 
how to use it, for example, how many tiers a region should have, how to 
separate each tier, and why?

This patch introduces several configuration parameters to separate tiers.  Do 
we have a guideline on how to use these parameters?  What kind tier separation 
will make the compaction scheduling efficient?  If the tiers are not set 
properly, could it make things even worse instead?

By the tier term, the existing compaction selection has just one tier.  Can we 
automatically skip the most recent storefile till we get a valid selection? By 
this way, we can have dynamical tiers which guarantees some valid selection?

 port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice 
 (not configurable by cf or dynamically)
 -

 Key: HBASE-7055
 URL: https://issues.apache.org/jira/browse/HBASE-7055
 Project: HBase
  Issue Type: Task
  Components: Compaction
Affects Versions: 0.96.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.96.0

 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, 
 HBASE-6371-v3-refactor-only-squashed.patch, 
 HBASE-6371-v4-refactor-only-squashed.patch, 
 HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, 
 HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch, 
 HBASE-7055-v4.patch


 There's divergence in the code :(
 See HBASE-6371 for details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6907) KeyValue equals and compareTo methods should match


[ 
https://issues.apache.org/jira/browse/HBASE-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540212#comment-13540212
 ] 

Ted Yu commented on HBASE-6907:
---

Over in https://reviews.apache.org/r/7591 (for HBASE-4676), memstoreTS is named 
MvccVersion

 KeyValue equals and compareTo methods should match
 --

 Key: HBASE-6907
 URL: https://issues.apache.org/jira/browse/HBASE-6907
 Project: HBase
  Issue Type: Bug
  Components: util
Reporter: Matt Corgan
Priority: Critical
 Fix For: 0.96.0


 KeyValue.KVComparator includes the memstoreTS when comparing, however the 
 KeyValue.equals() method ignores the memstoreTS.
 The Comparator interface has always specified that comparator return 0 when 
 equals would return true and vice versa.  Obeying that rule has been sort of 
 optional in the past, but Java 7 introduces a new default collection sorting 
 algorithm called Tim Sort which relies on that behavior.  
 http://bugs.sun.com/view_bug.do?bug_id=6804124
 Possible problem spots:
 * there's a Collections.sort(KeyValues) in 
 RedundantKVGenerator.generateTestKeyValues(..)
 * TestColumnSeeking compares two collections of KeyValues using the 
 containsAll method.  It is intentionally ignoring memstoreTS, so will need an 
 alternative method for comparing the two collections.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5776) HTableMultiplexer


[ 
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540222#comment-13540222
 ] 

Hudson commented on HBASE-5776:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #315 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/315/])
HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java


 HTableMultiplexer 
 --

 Key: HBASE-5776
 URL: https://issues.apache.org/jira/browse/HBASE-5776
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
 Fix For: 0.96.0

 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 
 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.1.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.1.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.2.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.2.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.3.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.4.patch, 
 ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, 
 HBASE-5776-trunk-V2.patch


 There is a known issue in HBase client that single slow/dead region server 
 could slow down the multiput operations across all the region servers. So the 
 HBase client will be as slow as the slowest region server in the cluster. 
  
 To solve this problem, HTableMultiplexer will separate the multiput 
 submitting threads with the flush threads, which means the multiput operation 
 will be a nonblocking operation. 
 The submitting thread will shard all the puts into different queues based on 
 its destination region server and return immediately. The flush threads will 
 flush these puts from each queue to its destination region server. 
 Currently the HTableMultiplexer only supports the put operation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7188) Move classes into hbase-client

2012-12-27 Thread Elliott Clark (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-7188:
-

Attachment: HBASE-7188-2.patch

Rebased patch.

Regarding having region server stuff in common: 
* Yeah mostly exceptions are in there since they are shared.  The client needs 
to be able to de-serialize them if they are thrown over the wire.
* BloomType should move as well but this is such a huge patch that I wanted to 
keep the renamig and moving to a minimum as keeping it rebased to trunk is a 
good deal of work.


Security: This can either be in common or client.  It's shared between the 
server and the client so I thought common but it can go either place.

 Move classes into hbase-client
 --

 Key: HBASE-7188
 URL: https://issues.apache.org/jira/browse/HBASE-7188
 Project: HBase
  Issue Type: Sub-task
  Components: Client, IPC/RPC
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 0.96.0

 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch, 
 HBASE-7188-2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7188) Move classes into hbase-client


[ 
https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540230#comment-13540230
 ] 

stack commented on HBASE-7188:
--

Can the shared exceptions be anonymized?  i.e. not particular to the 
regionserver and therefore not have to be in the regionserver subpackage?

Ok on bloomtype.

A 'generic' security in common is probably fine.

 Move classes into hbase-client
 --

 Key: HBASE-7188
 URL: https://issues.apache.org/jira/browse/HBASE-7188
 Project: HBase
  Issue Type: Sub-task
  Components: Client, IPC/RPC
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 0.96.0

 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch, 
 HBASE-7188-2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7410) [snapshots] add snapshot/clone/restore/export docs to ref guide


 [ 
https://issues.apache.org/jira/browse/HBASE-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-7410:
---

Attachment: HBASE-7410-v0.patch

Added some notes, that should allow to start using snapshot.

 [snapshots] add snapshot/clone/restore/export docs to ref guide
 ---

 Key: HBASE-7410
 URL: https://issues.apache.org/jira/browse/HBASE-7410
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, snapshots
Affects Versions: hbase-6055
Reporter: Jonathan Hsieh
Priority: Blocker
 Fix For: hbase-6055, 0.96.0

 Attachments: HBASE-7410-v0.patch


 This will include additions to the ref guide about the different operations 
 provided and how to use them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start

2012-12-27 Thread Jonathan Hsieh (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540242#comment-13540242
 ] 

Jonathan Hsieh commented on HBASE-7294:
---

looks good. +1.  Running through tests and will commit if they pass.

 Check for snapshot file cleaners on start
 -

 Key: HBASE-7294
 URL: https://issues.apache.org/jira/browse/HBASE-7294
 Project: HBase
  Issue Type: Sub-task
  Components: Client, master, regionserver, snapshots, Zookeeper
Affects Versions: hbase-6055
Reporter: Jesse Yates
Assignee: Matteo Bertozzi
 Fix For: hbase-6055, 0.96.0

 Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch, 
 HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch, 
 HBASE-7294-v6.patch


 Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to 
 ensure that any hfiles or hlogs (respectively) that are currently part of a 
 snapshot are not removed from their respective archive directories (.archive 
 and .oldlogs).
 From Matteo Bertozzi:
 {quote}
 currently the snapshot cleaner is not in hbase-default.xml
 and there's no warning/exception on snapshot/restore operation, if not 
 enabled.
 even if we add the cleaner to the hbase-default.xml how do we ensure that the 
 user doesn't remove it?
 Do we want to hardcode the cleaner at master startup?
 Do we want to add a check in snapshot/restore that throws an exception if the 
 cleaner is not enabled?
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7403) Online Merge

2012-12-27 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540257#comment-13540257
 ] 

Hadoop QA commented on HBASE-7403:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12562362/7403-trunkv5.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
10 warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 findbugs{color}.  The patch appears to introduce 3 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces lines longer than 
100

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.TestGlobalMemStoreSize
  org.apache.hadoop.hbase.mapreduce.TestRowCounter
  org.apache.hadoop.hbase.replication.TestMasterReplication
  org.apache.hadoop.hbase.TestFullLogReconstruction
  org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
  org.apache.hadoop.hbase.TestClusterBootOrder
  org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
  org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
  org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
  org.apache.hadoop.hbase.client.TestHCM
  org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.ipc.TestDelayedRpc
  org.apache.hadoop.hbase.mapreduce.TestImportExport

 {color:red}-1 core zombie tests{color}.  There are 5 zombie test(s):   
at 
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeAllFromMinorCompaction(TestHFileOutputFormat.java:693)
at 
org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingFirstRegion(TestHBaseFsck.java:1281)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/3722//console

This message is automatically generated.

 Online Merge
 

 Key: HBASE-7403
 URL: https://issues.apache.org/jira/browse/HBASE-7403
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
 Fix For: 0.96.0, 0.94.5

 Attachments: 7403-trunkv5.patch, hbase-7403-94v1.patch, 
 hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv5.patch, 
 merge region.pdf


 The feature of this online merge:
 1.Online,no necessary to disable table
 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
 3.Easy to call merege request, no need to input a long region name, only 
 encoded name enough
 4.No limit when operation, you don't need to care event like Server Dead, 
 Balance, Split, Disabing/Enabing table, no need to care whether you send a 
 wrong merge request, it alread have done for you
 5.Only little offline time for two merging regions
 We need merge in the following cases：
 1.Region hole or region overlap, can’t be fix by hbck
 2.Region become empty because of TTL and not reasonable Rowkey

[jira] [Updated] (HBASE-7384) Introducing waitForCondition function into test cases

2012-12-27 Thread Jeffrey Zhong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-7384:
-

Attachment: hbase-7384.patch

This patch is a baby step towards on flaky tests. Most flaky tests I found 
especially for cross platform tests are due to not long enough hard coded time 
out values. It would be much more pleasant if we can easily identify premature 
time out situations and adjust max time out values dynamically by a system 
property according to different environment needs. 

I migrated two test cases to use the new waitFor functions as an example. The 
new waitFor function will fully benefit us after we migrate all old test cases 
and use the new waitFor pattern for future test cases.

Thanks,
-Jeffrey


 Introducing waitForCondition function into test cases
 -

 Key: HBASE-7384
 URL: https://issues.apache.org/jira/browse/HBASE-7384
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
 Attachments: hbase-7384.patch, Waiter.java


 Recently I'm working on flaky test cases and found we have many places using 
 while loop and sleep to wait for a condition to be true. There are several 
 issues in existing ways:
 1) Many similar code doing the same thing
 2) When time out happens, different errors are reported without explicitly 
 indicating a time out situation
 3) When we want to increase the max timeout value to verify if a test case 
 fails due to a not-enough time out value, we have to recompile  redeploy code
 I propose to create a waitForCondition function as a test utility function 
 like the following:
 {code}
 public interface WaitCheck {
 public boolean Check() ;
 }
 public boolean waitForCondition(int timeOutInMilliSeconds, int 
 checkIntervalInMilliSeconds, WaitCheck s)
 throws InterruptedException {
 int multiplier = 1;
 String multiplierProp = System.getProperty(extremeWaitMultiplier);
 if(multiplierProp != null) {
 multiplier = Integer.parseInt(multiplierProp);
 if(multiplier  1) {
 LOG.warn(String.format(Invalid extremeWaitMultiplier 
 property value:%s. is ignored., multiplierProp));
 multiplier = 1;
 }
 }
 int timeElapsed = 0;
 while(timeElapsed  timeOutInMilliSeconds * multiplier) {
 if(s.Check()) {
 return true;
 }
 Thread.sleep(checkIntervalInMilliSeconds);
 timeElapsed += checkIntervalInMilliSeconds;
 }
 assertTrue(WaitForCondition failed due to time out( + 
 timeOutInMilliSeconds +  milliseconds expired),
 false);
 return false;
 }
 {code}
 By doing the above way, there are several advantages:
 1) Clearly report time out error when such situation happens
 2) Use System property extremeWaitMultiplier to increase max time out 
 dynamically for a quick verification
 3) Standardize current wait situations
 Pleas let me know what your thoughts on this.
 Thanks,
 -Jeffrey

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7384) Introducing waitForCondition function into test cases

2012-12-27 Thread Jeffrey Zhong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-7384:
-

Fix Version/s: 0.96.0
   Labels: test  (was: )
   Status: Patch Available  (was: Open)

 Introducing waitForCondition function into test cases
 -

 Key: HBASE-7384
 URL: https://issues.apache.org/jira/browse/HBASE-7384
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
  Labels: test
 Fix For: 0.96.0

 Attachments: hbase-7384.patch, Waiter.java


 Recently I'm working on flaky test cases and found we have many places using 
 while loop and sleep to wait for a condition to be true. There are several 
 issues in existing ways:
 1) Many similar code doing the same thing
 2) When time out happens, different errors are reported without explicitly 
 indicating a time out situation
 3) When we want to increase the max timeout value to verify if a test case 
 fails due to a not-enough time out value, we have to recompile  redeploy code
 I propose to create a waitForCondition function as a test utility function 
 like the following:
 {code}
 public interface WaitCheck {
 public boolean Check() ;
 }
 public boolean waitForCondition(int timeOutInMilliSeconds, int 
 checkIntervalInMilliSeconds, WaitCheck s)
 throws InterruptedException {
 int multiplier = 1;
 String multiplierProp = System.getProperty(extremeWaitMultiplier);
 if(multiplierProp != null) {
 multiplier = Integer.parseInt(multiplierProp);
 if(multiplier  1) {
 LOG.warn(String.format(Invalid extremeWaitMultiplier 
 property value:%s. is ignored., multiplierProp));
 multiplier = 1;
 }
 }
 int timeElapsed = 0;
 while(timeElapsed  timeOutInMilliSeconds * multiplier) {
 if(s.Check()) {
 return true;
 }
 Thread.sleep(checkIntervalInMilliSeconds);
 timeElapsed += checkIntervalInMilliSeconds;
 }
 assertTrue(WaitForCondition failed due to time out( + 
 timeOutInMilliSeconds +  milliseconds expired),
 false);
 return false;
 }
 {code}
 By doing the above way, there are several advantages:
 1) Clearly report time out error when such situation happens
 2) Use System property extremeWaitMultiplier to increase max time out 
 dynamically for a quick verification
 3) Standardize current wait situations
 Pleas let me know what your thoughts on this.
 Thanks,
 -Jeffrey

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7384) Introducing waitForCondition function into test cases

2012-12-27 Thread Jeffrey Zhong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-7384:
-

Attachment: hbase-7384_1.0.patch

 Introducing waitForCondition function into test cases
 -

 Key: HBASE-7384
 URL: https://issues.apache.org/jira/browse/HBASE-7384
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
  Labels: test
 Fix For: 0.96.0

 Attachments: hbase-7384_1.0.patch, hbase-7384.patch, Waiter.java


 Recently I'm working on flaky test cases and found we have many places using 
 while loop and sleep to wait for a condition to be true. There are several 
 issues in existing ways:
 1) Many similar code doing the same thing
 2) When time out happens, different errors are reported without explicitly 
 indicating a time out situation
 3) When we want to increase the max timeout value to verify if a test case 
 fails due to a not-enough time out value, we have to recompile  redeploy code
 I propose to create a waitForCondition function as a test utility function 
 like the following:
 {code}
 public interface WaitCheck {
 public boolean Check() ;
 }
 public boolean waitForCondition(int timeOutInMilliSeconds, int 
 checkIntervalInMilliSeconds, WaitCheck s)
 throws InterruptedException {
 int multiplier = 1;
 String multiplierProp = System.getProperty(extremeWaitMultiplier);
 if(multiplierProp != null) {
 multiplier = Integer.parseInt(multiplierProp);
 if(multiplier  1) {
 LOG.warn(String.format(Invalid extremeWaitMultiplier 
 property value:%s. is ignored., multiplierProp));
 multiplier = 1;
 }
 }
 int timeElapsed = 0;
 while(timeElapsed  timeOutInMilliSeconds * multiplier) {
 if(s.Check()) {
 return true;
 }
 Thread.sleep(checkIntervalInMilliSeconds);
 timeElapsed += checkIntervalInMilliSeconds;
 }
 assertTrue(WaitForCondition failed due to time out( + 
 timeOutInMilliSeconds +  milliseconds expired),
 false);
 return false;
 }
 {code}
 By doing the above way, there are several advantages:
 1) Clearly report time out error when such situation happens
 2) Use System property extremeWaitMultiplier to increase max time out 
 dynamically for a quick verification
 3) Standardize current wait situations
 Pleas let me know what your thoughts on this.
 Thanks,
 -Jeffrey

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed

Zhong Deyin created HBASE-7445:
--

 Summary: Hbase cluster is unavailable while the regionserver that 
Meta table deployed crashed
 Key: HBASE-7445
 URL: https://issues.apache.org/jira/browse/HBASE-7445
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment, regionserver
Affects Versions: 0.94.1
 Environment: Hadoop 0.20.2-cdh3u3
Hbase 0.94.1
Reporter: Zhong Deyin
 Fix For: 0.94.1


while the regionserver that META table deployed crashed, the .META. table can't 
migrate to other available regionservers. Then the region spliting, can't find 
META table, cause the whole cluster is unavailable.
Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler
{code}
  // Carrying meta
  if (isCarryingMeta()) {
LOG.info(Server  + serverName +  was carrying META. Trying to 
assign.);
this.services.getAssignmentManager().
  regionOffline(HRegionInfo.FIRST_META_REGIONINFO);
this.services.getAssignmentManager().assignMeta();
  }
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed


 [ 
https://issues.apache.org/jira/browse/HBASE-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhong Deyin updated HBASE-7445:
---

Fix Version/s: (was: 0.94.1)

 Hbase cluster is unavailable while the regionserver that Meta table deployed 
 crashed
 

 Key: HBASE-7445
 URL: https://issues.apache.org/jira/browse/HBASE-7445
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment, regionserver
Affects Versions: 0.94.1
 Environment: Hadoop 0.20.2-cdh3u3
 Hbase 0.94.1
Reporter: Zhong Deyin
  Labels: patch
   Original Estimate: 336h
  Remaining Estimate: 336h

 while the regionserver that META table deployed crashed, the .META. table 
 can't migrate to other available regionservers. Then the region spliting, 
 can't find META table, cause the whole cluster is unavailable.
 Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler
 {code}
   // Carrying meta
   if (isCarryingMeta()) {
 LOG.info(Server  + serverName +  was carrying META. Trying to 
 assign.);
 this.services.getAssignmentManager().
   regionOffline(HRegionInfo.FIRST_META_REGIONINFO);
 this.services.getAssignmentManager().assignMeta();
   }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7384) Introducing waitForCondition function into test cases

2012-12-27 Thread Enis Soztutar (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540276#comment-13540276
 ] 

Enis Soztutar commented on HBASE-7384:
--

This tool is useful, but as Nick pointed out, we should really be using 
CountDownLatch'es for test synchronization. The problem, however, is that most 
of the time, the latch release should happen from the main code, and there is 
no generic injection mechanism yet to do this from the tests cleanly. For 
example the patch in HBASE-5494 introduces a class called InjectionHandler for 
doing this, but not sure if it is the best interface. 

Above is out of the scope for this isssue, but I think we should keep that in 
mind. 

 Introducing waitForCondition function into test cases
 -

 Key: HBASE-7384
 URL: https://issues.apache.org/jira/browse/HBASE-7384
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
  Labels: test
 Fix For: 0.96.0

 Attachments: hbase-7384_1.0.patch, hbase-7384.patch, Waiter.java


 Recently I'm working on flaky test cases and found we have many places using 
 while loop and sleep to wait for a condition to be true. There are several 
 issues in existing ways:
 1) Many similar code doing the same thing
 2) When time out happens, different errors are reported without explicitly 
 indicating a time out situation
 3) When we want to increase the max timeout value to verify if a test case 
 fails due to a not-enough time out value, we have to recompile  redeploy code
 I propose to create a waitForCondition function as a test utility function 
 like the following:
 {code}
 public interface WaitCheck {
 public boolean Check() ;
 }
 public boolean waitForCondition(int timeOutInMilliSeconds, int 
 checkIntervalInMilliSeconds, WaitCheck s)
 throws InterruptedException {
 int multiplier = 1;
 String multiplierProp = System.getProperty(extremeWaitMultiplier);
 if(multiplierProp != null) {
 multiplier = Integer.parseInt(multiplierProp);
 if(multiplier  1) {
 LOG.warn(String.format(Invalid extremeWaitMultiplier 
 property value:%s. is ignored., multiplierProp));
 multiplier = 1;
 }
 }
 int timeElapsed = 0;
 while(timeElapsed  timeOutInMilliSeconds * multiplier) {
 if(s.Check()) {
 return true;
 }
 Thread.sleep(checkIntervalInMilliSeconds);
 timeElapsed += checkIntervalInMilliSeconds;
 }
 assertTrue(WaitForCondition failed due to time out( + 
 timeOutInMilliSeconds +  milliseconds expired),
 false);
 return false;
 }
 {code}
 By doing the above way, there are several advantages:
 1) Clearly report time out error when such situation happens
 2) Use System property extremeWaitMultiplier to increase max time out 
 dynamically for a quick verification
 3) Standardize current wait situations
 Pleas let me know what your thoughts on this.
 Thanks,
 -Jeffrey

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed


 [ 
https://issues.apache.org/jira/browse/HBASE-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhong Deyin resolved HBASE-7445.


Resolution: Fixed

modify class org.apache.hadoop.hbase.master.handler.ServerShutdownHandler, 
change the process method, replace code 
this.services.getAssignmentManager().assignMeta() to 
assignMetaWithRetries(),then meta will try 10 times while the regionservers 
crashed.
{code}
  // Carrying meta
  if (isCarryingMeta()) {
LOG.info(Server  + serverName +
   was carrying META. Trying to assign.);
this.services.getAssignmentManager().
  regionOffline(HRegionInfo.FIRST_META_REGIONINFO);
//this.services.getAssignmentManager().assignMeta();
assignMetaWithRetries();
  }
{code}
Add method assignMetaWithRetries, code of assignMetaWithRetries method as 
follows:
{code}
 private void assignMetaWithRetries() throws IOException{
  int iTimes = this.server.getConfiguration().getInt(
hbase.catalog.verification.retries, 10);

long waitTime = this.server.getConfiguration().getLong(
hbase.catalog.verification.timeout, 1000);

int iFlag = 0;
LOG.info(TEST START);
while (true) {
  try {
   // verifyAndAssignRoot();
this.services.getAssignmentManager().assignMeta();
break;
  } catch (Exception e) {
if (iFlag = iTimes) {
  this.server.abort(  test ming  assginMeta failed 
after + iTimes
  +  times retries, aborting, e);
  throw new IOException(Aborting, e);
}
try {
  Thread.sleep(waitTime);
} catch (InterruptedException e1) {
  LOG.warn(Interrupted when is the thread sleep, e1);
  Thread.currentThread().interrupt();
  throw new IOException(Interrupted, e1);
}
iFlag++;
  }
} 
LOG.info(TEST END HBASE);
  }
{code}

 Hbase cluster is unavailable while the regionserver that Meta table deployed 
 crashed
 

 Key: HBASE-7445
 URL: https://issues.apache.org/jira/browse/HBASE-7445
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment, regionserver
Affects Versions: 0.94.1
 Environment: Hadoop 0.20.2-cdh3u3
 Hbase 0.94.1
Reporter: Zhong Deyin
  Labels: patch
   Original Estimate: 336h
  Remaining Estimate: 336h

 while the regionserver that META table deployed crashed, the .META. table 
 can't migrate to other available regionservers. Then the region spliting, 
 can't find META table, cause the whole cluster is unavailable.
 Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler
 {code}
   // Carrying meta
   if (isCarryingMeta()) {
 LOG.info(Server  + serverName +  was carrying META. Trying to 
 assign.);
 this.services.getAssignmentManager().
   regionOffline(HRegionInfo.FIRST_META_REGIONINFO);
 this.services.getAssignmentManager().assignMeta();
   }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed


 [ 
https://issues.apache.org/jira/browse/HBASE-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhong Deyin reopened HBASE-7445:



 Hbase cluster is unavailable while the regionserver that Meta table deployed 
 crashed
 

 Key: HBASE-7445
 URL: https://issues.apache.org/jira/browse/HBASE-7445
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment, regionserver
Affects Versions: 0.94.1
 Environment: Hadoop 0.20.2-cdh3u3
 Hbase 0.94.1
Reporter: Zhong Deyin
  Labels: patch
   Original Estimate: 336h
  Remaining Estimate: 336h

 while the regionserver that META table deployed crashed, the .META. table 
 can't migrate to other available regionservers. Then the region spliting, 
 can't find META table, cause the whole cluster is unavailable.
 Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler
 {code}
   // Carrying meta
   if (isCarryingMeta()) {
 LOG.info(Server  + serverName +  was carrying META. Trying to 
 assign.);
 this.services.getAssignmentManager().
   regionOffline(HRegionInfo.FIRST_META_REGIONINFO);
 this.services.getAssignmentManager().assignMeta();
   }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine

2012-12-27 Thread Devaraj Das (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540306#comment-13540306
 ] 

Devaraj Das commented on HBASE-5705:


[~stack], yes. The idea was to have the server decide how to best handle the 
rpc method call based on the client's protocol version. Though the server 
currently ignores the version, this is something that can be used to implement 
versioning. Please have a look at ProtobufRpcServerEngine.java around the area 
with the comment TODO: use the clientVersion... Other than that, HBASE-6521 
should address versioning in a more complete fashion (and I guess we should 
unify all the versioning stories there).

 Introduce Protocol Buffer RPC engine
 

 Key: HBASE-5705
 URL: https://issues.apache.org/jira/browse/HBASE-5705
 Project: HBase
  Issue Type: Sub-task
  Components: IPC/RPC, master, migration, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
 Fix For: 0.96.0

 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch, 
 5705-2.3.patch


 Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB 
 aware can be made to go through this RPC engine. The approach, in my current 
 thinking, would be similar to HADOOP-7773.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540307#comment-13540307
 ] 

Ted Yu commented on HBASE-3776:
---

I already gave my +1.
One more +1 is needed.

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7446) Make the reference guide utf8 rather than 8859


 [ 
https://issues.apache.org/jira/browse/HBASE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-7446:
-

Attachment: utf8.txt

Here is a patch that seems to do the right thing.

 Make the reference guide utf8 rather than 8859
 --

 Key: HBASE-7446
 URL: https://issues.apache.org/jira/browse/HBASE-7446
 Project: HBase
  Issue Type: Bug
Reporter: stack
 Attachments: utf8.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540303#comment-13540303
 ] 

Anoop Sam John commented on HBASE-3776:
---

@Ted, @stack  - Is it good to go?

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-7446) Make the reference guide utf8 rather than 8859


 [ 
https://issues.apache.org/jira/browse/HBASE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-7446.
--

   Resolution: Fixed
Fix Version/s: 0.96.0
 Assignee: stack

Committed to trunk.

 Make the reference guide utf8 rather than 8859
 --

 Key: HBASE-7446
 URL: https://issues.apache.org/jira/browse/HBASE-7446
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Fix For: 0.96.0

 Attachments: utf8.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7446) Make the reference guide utf8 rather than 8859

stack created HBASE-7446:


 Summary: Make the reference guide utf8 rather than 8859
 Key: HBASE-7446
 URL: https://issues.apache.org/jira/browse/HBASE-7446
 Project: HBase
  Issue Type: Bug
Reporter: stack
 Attachments: utf8.txt



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6132) ColumnCountGetFilter PageFilter not working with FilterList


[ 
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540304#comment-13540304
 ] 

Anoop Sam John commented on HBASE-6132:
---

Devs
I feel the attached test case reveals the problem with FilterList. If all 
agree I can make the change and submit a patch.
[~saint@gmail.com] Pls check the test case once

 ColumnCountGetFilter  PageFilter not working with FilterList
 -

 Key: HBASE-6132
 URL: https://issues.apache.org/jira/browse/HBASE-6132
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 0.92.0, 0.92.1, 0.94.0
 Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0, 
 zookeeper 3.4.3
Reporter: Benjamin Kim
Assignee: Anoop Sam John
 Attachments: TestColumnCountGetFilter.java


 Thanks to Anoop and Ramkrishna, here's what we found with FilterList
 If I use FilterList to include ColumnCountGetFilter among other filters, the 
 returning Result has no keyvalues.
 This problem seems to occur when specified column count is less then actual 
 number of existing columns.
 Also same problem arises with PageFilter
 Following is the code of the problem:
 {code}
 Configuration conf = HBaseConfiguration.create();
 HTable table = new HTable(conf, test);
 Get get = new Get(Bytes.toBytes(test1));
 FilterList filterList = new FilterList();
 filterList.addFilter(new ColumnCountGetFilter(100));   
 get.setFilter(filterList);
 Result r = table.get(get);
 System.out.println(r.size()); // prints zero
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7447) Occasional failures of TestSplitLogWorker#testAcquireTaskAtStartup

Andrew Purtell created HBASE-7447:
-

 Summary: Occasional failures of 
TestSplitLogWorker#testAcquireTaskAtStartup
 Key: HBASE-7447
 URL: https://issues.apache.org/jira/browse/HBASE-7447
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.94.4
Reporter: Andrew Purtell


I see occasional failures (1 out of ~20) on branch 0.94 of 
TestSplitLogWorker#testAcquireTaskAtStartup

{noformat}
Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.737 sec  
FAILURE!
Failed tests:

testAcquireTaskAtStartup(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker):
 ctr=0, oldval=0, newval=1
{noformat}

Bisecting commits on 0.94 branch, using (up to) 50 repetitions to include or 
exclude, I get:

{noformat}
8cc5a5f481aeaf6debfa95110c4392d9c3ea20ca is the first bad commit
commit 8cc5a5f481aeaf6debfa95110c4392d9c3ea20ca
HBASE-7172 TestSplitLogManager.testVanishingTaskZNode() fails when run 
individually and is flaky
git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.94@1414975 
13f79535-47bb-0310-9956-ffa450edef68
{noformat}

Will try again to see if the search ends in the same place.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540311#comment-13540311
 ] 

Lars Hofhansl commented on HBASE-3776:
--

Patch looks good to me +1
I think we should have this in 0.94 as well.

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row

[
https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540312#comment-13540312
]

Anoop Sam John commented on HBASE-5664:
---

This is needed in the Secondary index solution
The scenario is as below
Having 2 conditions in FilterList with MUST_PASS_ALL
One SCVF on column cf1:c1 and another scvf on column cf1:c2
Suppose there is only one index on column cf1:c1
Now based on the data got from the index table we will seek to the exact row
where cf1:c1=?. But dont know what is the value of cf1:c2 there in that row.
Now the second filter may filter out this row.
As per the logic in HRegion.nextInternal() it will start a loop there and
taking next row onwards till finding a row satisfying both conditions

Now as the condition in FL is MUST_PASS_ALL, there is no point in continuing
with all the rows. The next row satisfying the condition on column cf1:c1 , we
can know from the index data and so we can make a seek to that row directly and
check that.

Now there is no way to do so. It will be good to provide a CP hook for this so
that within that hook I can reseek to the exact row key.

CP hooks in Scan flow for fast forward when filter filters out a row

Key: HBASE-5664
URL: https://issues.apache.org/jira/browse/HBASE-5664
Project: HBase
Issue Type: Improvement
Components: Coprocessors
Affects Versions: 0.92.1
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.96.0

In HRegion.nextInternal(int limit, String metric)
We have while(true) loop so as to fetch a next result which satisfies
filter condition. When Filter filters out the current fetched row we call
nextRow(byte [] currentRow) before going with the next row.
{code}
if (results.isEmpty() || filterRow()) {
// this seems like a redundant step - we already consumed the row
// there're no left overs.
// the reasons for calling this method are:
// 1. reset the filters.
// 2. provide a hook to fast forward the row (used by subclasses)
nextRow(currentRow);
{code}
// 2. provide a hook to fast forward the row (used by subclasses)
We can provide same feature of fast forward support for the CP also.

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540315#comment-13540315
 ] 

Anoop Sam John commented on HBASE-3776:
---

Lars
I can give a patch for 94 version also. Will upload tonight..

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540318#comment-13540318
 ] 

Ted Yu commented on HBASE-3776:
---

Integrated to trunk.

Thanks for the patch, Anoop.

Thanks for the review, Lars.

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540322#comment-13540322
 ] 

Hudson commented on HBASE-3776:
---

Integrated in HBase-TRUNK #3664 (See 
[https://builds.apache.org/job/HBase-TRUNK/3664/])
HBASE-3776 Add Bloom Filter Support to HFileOutputFormat (Anoop Sam John) 
(Revision 1426406)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java


 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7446) Make the reference guide utf8 rather than 8859


[ 
https://issues.apache.org/jira/browse/HBASE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540323#comment-13540323
 ] 

Hudson commented on HBASE-7446:
---

Integrated in HBase-TRUNK #3664 (See 
[https://builds.apache.org/job/HBase-TRUNK/3664/])
hbase-7446 Make the reference guide utf8 rather than 8859 (Revision 1426400)

 Result = FAILURE

 Make the reference guide utf8 rather than 8859
 --

 Key: HBASE-7446
 URL: https://issues.apache.org/jira/browse/HBASE-7446
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Fix For: 0.96.0

 Attachments: utf8.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7448) Add tags to Cell API

Andrew Purtell created HBASE-7448:
-

 Summary: Add tags to Cell API
 Key: HBASE-7448
 URL: https://issues.apache.org/jira/browse/HBASE-7448
 Project: HBase
  Issue Type: Task
Affects Versions: 0.96.0
Reporter: Andrew Purtell


See 
https://issues.apache.org/jira/browse/HBASE-7233?focusedCommentId=13531619page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13531619
 and nearby comments on HBASE-7233 for background.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


 [ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-3776:
--

Attachment: 3776-trunk.addendum

Addendum for trunk that fixes reference to BloomType

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: 3776-trunk.addendum, HBASE-3776.patch, 
 HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package


[ 
https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540332#comment-13540332
 ] 

stack commented on HBASE-7224:
--

Any chance of a review for this blocker against 0.96 issue?

 Remove references to Writable in the ipc package
 

 Key: HBASE-7224
 URL: https://issues.apache.org/jira/browse/HBASE-7224
 Project: HBase
  Issue Type: Sub-task
  Components: IPC/RPC, Protobufs
Reporter: Devaraj Das
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, 
 7224v4.txt, 7224v4.txt, purge_more_writables.txt


 I see references to Writable in the ipc package, most notably in the 
 Invocation class. This class is not being used that much in the core ipc 
 package but used in the coprocessor protocol implementations (there are some 
 coprocessor protocols that are Writable based still). This jira is to track 
 removing those references and the Invocation class (once HBASE-6895 is 
 resolved).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7448) Add tags to Cell API


 [ 
https://issues.apache.org/jira/browse/HBASE-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-7448:
--

Attachment: 7448_todo.txt

Attached as 7448_todo.txt just an idea for the interface, implementation TODO. 
Moves Pair from hbase-server to hbase-common. 

 Add tags to Cell API
 

 Key: HBASE-7448
 URL: https://issues.apache.org/jira/browse/HBASE-7448
 Project: HBase
  Issue Type: Task
Affects Versions: 0.96.0
Reporter: Andrew Purtell
 Attachments: 7448_todo.txt


 See 
 https://issues.apache.org/jira/browse/HBASE-7233?focusedCommentId=13531619page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13531619
  and nearby comments on HBASE-7233 for background.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row


[ 
https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540334#comment-13540334
 ] 

Lars Hofhansl commented on HBASE-5664:
--

As long as the reseek is optional it's fine, I think.
A (re)seek() is potentially *much* slower than just a next(). So when the next 
row is only a few next()'s away replacing that with (re)seek() can have 
disastrous performance effects (as seen in HBASE-6577).


 CP hooks in Scan flow for fast forward when filter filters out a row
 

 Key: HBASE-5664
 URL: https://issues.apache.org/jira/browse/HBASE-5664
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors
Affects Versions: 0.92.1
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 0.96.0


 In HRegion.nextInternal(int limit, String metric)
   We have while(true) loop so as to fetch a next result which satisfies 
 filter condition. When Filter filters out the current fetched row we call 
 nextRow(byte [] currentRow) before going with the next row.
 {code}
 if (results.isEmpty() || filterRow()) {
 // this seems like a redundant step - we already consumed the row
 // there're no left overs.
 // the reasons for calling this method are:
 // 1. reset the filters.
 // 2. provide a hook to fast forward the row (used by subclasses)
 nextRow(currentRow);
 {code}
 // 2. provide a hook to fast forward the row (used by subclasses)
 We can provide same feature of fast forward support for the CP also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


 [ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-3776:
-

Status: Open  (was: Patch Available)

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: 3776-trunk.addendum, HBASE-3776.patch, 
 HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


 [ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-3776:
-

Attachment: 3776-0.94.txt

Here's a 0.94 patch.

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: 3776-0.94.txt, 3776-trunk.addendum, HBASE-3776.patch, 
 HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package

2012-12-27 Thread Devaraj Das (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540336#comment-13540336
 ] 

Devaraj Das commented on HBASE-7224:


I'll make a pass over this soon.

 Remove references to Writable in the ipc package
 

 Key: HBASE-7224
 URL: https://issues.apache.org/jira/browse/HBASE-7224
 Project: HBase
  Issue Type: Sub-task
  Components: IPC/RPC, Protobufs
Reporter: Devaraj Das
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, 
 7224v4.txt, 7224v4.txt, purge_more_writables.txt


 I see references to Writable in the ipc package, most notably in the 
 Invocation class. This class is not being used that much in the core ipc 
 package but used in the coprocessor protocol implementations (there are some 
 coprocessor protocols that are Writable based still). This jira is to track 
 removing those references and the Invocation class (once HBASE-6895 is 
 resolved).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540338#comment-13540338
 ] 

Lars Hofhansl commented on HBASE-3776:
--

Opps @Anop. Just saw you latest comment now. Anyway, please review the 0.94 
patch I attached.

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce
Reporter: Nicolas Spiegelberg
Assignee: Anoop Sam John
Priority: Critical
  Labels: hbase
 Fix For: 0.96.0

 Attachments: 3776-0.94.txt, 3776-trunk.addendum, HBASE-3776.patch, 
 HBASE-3776_v2.patch


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package