date:20150810

[jira] [Updated] (HBASE-13062) Add documentation coverage for configuring dns server with thrift and rest gateways

2015-08-10 Thread Misty Stanley-Jones (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-13062:

Attachment: HBASE-13062-v1.patch

How is this?

 Add documentation coverage for configuring dns server with thrift and rest 
 gateways
 ---

 Key: HBASE-13062
 URL: https://issues.apache.org/jira/browse/HBASE-13062
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Srikanth Srungarapu
Assignee: Misty Stanley-Jones
Priority: Minor
 Attachments: HBASE-13062-v1.patch, HBASE-13062.patch


 Currently, the documentation doesn't cover about configuring DNS with thrift 
 or rest gateways, though code base does provide provision for doing so. The 
 following parameters are being used for accomplishing the same.
 For REST:
 * hbase.rest.dns.interface
 * hbase.rest.dns.nameserver
 For Thrift:
 * hbase.thrift.dns.interface
 * hbase.thrift.dns.nameserver



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14202) Reduce garbage we create

2015-08-10 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681223#comment-14681223
 ] 

Anoop Sam John commented on HBASE-14202:


Thanks Stack.  Will commit later tonight once QA passes.
bq.The Object/int pair is ugly 
Agree.. Bit odd it is.. But it helps us a lot wrt garbage we create.  :-)
This kind of more fine tuning sub tasks will come in under this Jira. We are 
doing more profiling stuff wrt CPU time and object/memory usage.

 Reduce garbage we create
 

 Key: HBASE-14202
 URL: https://issues.apache.org/jira/browse/HBASE-14202
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Affects Versions: 2.0.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0

 Attachments: HBASE-14202.patch


 2 optimizations wrt no# short living objects we create
 1. IOEngine#read call to read from L2 cache is always creating a Pair object 
 to return the BB and MemoryType. We can avoid this by making the read API to 
 return a Cacheable. Pass the CacheableDeserializer, to be used, also to the 
 read API. Setter for MemoryType is already there in Cacheable interface.
 2. ByteBuff#asSubByteBuffer(int, int, Pair)  avoids Pair object creation 
 every time as we pass the shared Pair object. Still as pair can take only 
 Objects, the primitive int has to be boxed into an Integer object every time. 
 This can be avoided by creating a new Pair type which is a pair of an Object 
 and a primitive int.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13907) Document how to deploy a coprocessor

2015-08-10 Thread Misty Stanley-Jones (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-13907:

Attachment: HBASE-13907-v3.patch

This addresses Andrew's point. Sean's point #3 is still an open issue. Can 
anyone define the behavior?

 Document how to deploy a coprocessor
 

 Key: HBASE-13907
 URL: https://issues.apache.org/jira/browse/HBASE-13907
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Attachments: HBASE-13907-1.patch, HBASE-13907-2.patch, 
 HBASE-13907-v3.patch, HBASE-13907.patch


 Capture this information:
  Where are the dependencies located for these classes? Is there a path on 
  HDFS or local disk that dependencies need to be placed so that each 
  RegionServer has access to them?
 It is suggested to bundle them as a single jar so that RS can load the whole 
 jar and resolve dependencies. If you are not able to do that, you need place 
 the dependencies in regionservers class path so that they are loaded during 
 RS startup. Do either of these options work for you? Btw, you can load the 
 coprocessors/filters into path specified by hbase.dynamic.jars.dir [1], so 
 that they are loaded dynamically by regionservers when the class is accessed 
 (or you can place them in the RS class path too, so that they are loaded 
 during RS JVM startup).
  How would one deploy these using an automated system? 
  (puppet/chef/ansible/etc)
 You can probably use these tools to automate shipping the jars to above 
 locations?
  Tests our developers have done suggest that simply disabling a coprocessor, 
  replacing the jar with a different version, and enabling the coprocessor 
  again does not load the newest version. With that in mind how does one know 
  which version is currently deployed and enabled without resorting to 
  parsing `hbase shell` output or restarting hbase?
 Actually this is a design issue with current classloader. You can't reload a 
 class in a JVM unless you delete all the current references to it. Since the 
 current JVM (classloader) has reference to it, you can't overwrite it unless 
 you kill the JVM, which is equivalent to restarting it. So you still have the 
 older class loaded in place. For this to work, classloader design should be 
 changed. If it works for you, you can rename the coprocessor class name and 
 the new version of jar and RS loads it properly.
  Where does logging go, and how does one access it? Does logging need to be 
  configured in a certain way?
 Can you please specify which logging you are referring to?
  Where is a good location to place configuration files?
 Same as above, are these hbase configs or something else? If hbase configs, 
 are these gateway configs/server side? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14186) Read mvcc vlong optimization

2015-08-10 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681230#comment-14681230
 ] 

Anoop Sam John commented on HBASE-14186:


Yes Andy. Not direct patch substitute.  Sorry I missed this backport stuff in 
the middle of some thing else. Bit busy with personal stuff this whole week 
too.. I will close this jira as of now and will open a backport one for 1.x.  
If any one up for a backport jira in 2 days time I can keep it open.

 Read mvcc vlong optimization
 

 Key: HBASE-14186
 URL: https://issues.apache.org/jira/browse/HBASE-14186
 Project: HBase
  Issue Type: Sub-task
  Components: Performance, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0

 Attachments: HBASE-14186.patch


 {code}
 for (int idx = 0; idx  remaining; idx++) {
   byte b = blockBuffer.getByteAfterPosition(offsetFromPos + idx);
   i = i  8;
   i = i | (b  0xFF);
 }
 {code}
 Doing the read as in case of BIG_ENDIAN.
 After HBASE-12600, we tend to keep the mvcc and so byte by byte read looks 
 eating up lot of CPU time. (In my test HFileReaderImpl#_readMvccVersion comes 
 on top in terms of hot methods). We can optimize here by reading 4 or 2 bytes 
 in one shot when the length of the vlong is more than 4 bytes. We will in 
 turn use UnsafeAccess methods which handles ENDIAN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.

2015-08-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681267#comment-14681267
 ] 

Hudson commented on HBASE-5878:
---

FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1026 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1026/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from 
Hadoop-2. (apurtell: rev b69569f512068d795199310ce662ab381bb6b6b7)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Revert HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream 
from Hadoop-2. (apurtell: rev fabfb423f9cf48ddd52e9583ca6664f42349)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java


 Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
 ---

 Key: HBASE-5878
 URL: https://issues.apache.org/jira/browse/HBASE-5878
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch, 
 HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, 
 HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch, 
 HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch


 SequencFileLogReader: 
 Currently Hbase using getFileLength api from DFSInputStream class by 
 reflection. DFSInputStream is not exposed as public. So, this may change in 
 future. Now HDFS exposed HdfsDataInputStream as public API.
 We can make use of it, when we are not able to find the getFileLength api 
 from DFSInputStream as a else condition. So, that we will not have any sudden 
 surprise like we are facing today.
 Also,  it is just logging one warn message and proceeding if it throws any 
 exception while getting the length. I think we can re-throw the exception 
 because there is no point in continuing with dataloss.
 {code}
 long adjust = 0;
   try {
 Field fIn = FilterInputStream.class.getDeclaredField(in);
 fIn.setAccessible(true);
 Object realIn = fIn.get(this.in);
 // In hadoop 0.22, DFSInputStream is a standalone class.  Before 
 this,
 // it was an inner class of DFSClient.
 if (realIn.getClass().getName().endsWith(DFSInputStream)) {
   Method getFileLength = realIn.getClass().
 getDeclaredMethod(getFileLength, new Class? []{});
   getFileLength.setAccessible(true);
   long realLength = ((Long)getFileLength.
 invoke(realIn, new Object []{})).longValue();
   assert(realLength = this.length);
   adjust = realLength - this.length;
 } else {
   LOG.info(Input stream class:  + realIn.getClass().getName() +
   , not adjusting length);
 }
   } catch(Exception e) {
 SequenceFileLogReader.LOG.warn(
   Error while trying to get accurate file length.   +
   Truncation / data loss may occur if RegionServers die., e);
   }
   return adjust + super.getPos();
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14194) Undeprecate methods in ThriftServerRunner.HBaseHandler

2015-08-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681268#comment-14681268
 ] 

Hudson commented on HBASE-14194:


FAILURE: Integrated in HBase-1.3 #100 (See 
[https://builds.apache.org/job/HBase-1.3/100/])
HBASE-14194 Undeprecate methods in ThriftServerRunner.HBaseHandler (apurtell: 
rev c07eb21e4be74cac4756cf44331269257ac56daa)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java


 Undeprecate methods in ThriftServerRunner.HBaseHandler
 --

 Key: HBASE-14194
 URL: https://issues.apache.org/jira/browse/HBASE-14194
 Project: HBase
  Issue Type: Improvement
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Trivial
 Fix For: 2.0.0, 1.2.0, 1.3.0

 Attachments: HBASE-14194.patch


 The methods {{get}}, {{getVer}}, {{getVerTs}}, {{atomicIncrement}} were 
 deprecated back in HBASE-1304. My guess is this was because it wasn't 
 distinguishing between column family and column qualifier but I'm not sure. 
 Either way it's been in there for six years without documentation or a 
 deprecation at the interface level so it adds to my confusion and I'll attach 
 a patch to remove the deprecations.
 I guess at one point the whole old Thrift server will be deprecated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14200) Separate RegionReplica subtests of TestStochasticLoadBalancer into TestStochasticLoadBalancer2

2015-08-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14200:
---
Summary: Separate RegionReplica subtests of TestStochasticLoadBalancer into 
TestStochasticLoadBalancer2  (was: Separate some subtests of 
TestStochasticLoadBalancer into TestStochasticLoadBalancer2)

 Separate RegionReplica subtests of TestStochasticLoadBalancer into 
 TestStochasticLoadBalancer2
 --

 Key: HBASE-14200
 URL: https://issues.apache.org/jira/browse/HBASE-14200
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Fix For: 2.0.0, 1.3.0

 Attachments: 14200-v1.txt, 14200-v2.txt


 More and more functionality is added to StochasticLoadBalancer , making 
 TestStochasticLoadBalancer run longer.
 From 
 https://builds.apache.org/job/PreCommit-HBASE-Build/15011/testReport/org.apache.hadoop.hbase.master.balancer/TestStochasticLoadBalancer/
  where total runtime was 14 min, here are the longest subtests:
 testRegionReplicasOnLargeCluster: 1 min 34 sec
 testRegionReplicasOnMidCluster: 1 min 31 sec
 testRegionReplicasOnMidClusterHighReplication: 2 min
 testRegionReplicationOnMidClusterReplicationGreaterThanNumNodes: 2 min 25 sec
 This issue is to separate out the above subtests into 
 TestStochasticLoadBalancer2, giving each of the tests around 7 min runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14190) Assign system tables ahead of user region assignment

2015-08-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14190:
---
Attachment: 14190-v5.txt

Patch v5 adds an assertion in TestMasterFailover#testSimpleMasterFailover for 
namespace table which is assigned before active master initialization finishes.

 Assign system tables ahead of user region assignment
 

 Key: HBASE-14190
 URL: https://issues.apache.org/jira/browse/HBASE-14190
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Attachments: 14190-v1.txt, 14190-v2.txt, 14190-v3.txt, 14190-v3.txt, 
 14190-v3.txt, 14190-v4.txt, 14190-v5.txt


 Currently the namespace table region is assigned like user regions.
 I spent several hours working with a customer where master couldn't finish 
 initialization.
 Even though master was restarted quite a few times, it went down with the 
 following:
 {code}
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Master server abort: loaded coprocessors are: []
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Unhandled exception. Starting shutdown.
 java.io.IOException: Timedout 30ms waiting for namespace table to be 
 assigned
   at 
 org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
   at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
   at java.lang.Thread.run(Thread.java:744)
 {code}
 During previous run(s), namespace table was created, hence leaving an entry 
 in hbase:meta.
 The following if block in TableNamespaceManager#start() was skipped:
 {code}
 if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
   TableName.NAMESPACE_TABLE_NAME)) {
 {code}
 TableNamespaceManager#start() spins, waiting for namespace region to be 
 assigned.
 There was issue in master assigning user regions.
 We tried issuing 'assign' command from hbase shell which didn't work because 
 of the following check in MasterRpcServices#assignRegion():
 {code}
   master.checkInitialized();
 {code}
 This scenario can be avoided if we assign hbase:namespace table after 
 hbase:meta is assigned but before user table region assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14190) Assign system tables ahead of user region assignment

2015-08-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14190:
---
Attachment: (was: 14190-v3.txt)

 Assign system tables ahead of user region assignment
 

 Key: HBASE-14190
 URL: https://issues.apache.org/jira/browse/HBASE-14190
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Attachments: 14190-v1.txt, 14190-v2.txt, 14190-v3.txt, 14190-v3.txt, 
 14190-v4.txt, 14190-v5.txt


 Currently the namespace table region is assigned like user regions.
 I spent several hours working with a customer where master couldn't finish 
 initialization.
 Even though master was restarted quite a few times, it went down with the 
 following:
 {code}
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Master server abort: loaded coprocessors are: []
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Unhandled exception. Starting shutdown.
 java.io.IOException: Timedout 30ms waiting for namespace table to be 
 assigned
   at 
 org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
   at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
   at java.lang.Thread.run(Thread.java:744)
 {code}
 During previous run(s), namespace table was created, hence leaving an entry 
 in hbase:meta.
 The following if block in TableNamespaceManager#start() was skipped:
 {code}
 if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
   TableName.NAMESPACE_TABLE_NAME)) {
 {code}
 TableNamespaceManager#start() spins, waiting for namespace region to be 
 assigned.
 There was issue in master assigning user regions.
 We tried issuing 'assign' command from hbase shell which didn't work because 
 of the following check in MasterRpcServices#assignRegion():
 {code}
   master.checkInitialized();
 {code}
 This scenario can be avoided if we assign hbase:namespace table after 
 hbase:meta is assigned but before user table region assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.

2015-08-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680307#comment-14680307
 ] 

Hadoop QA commented on HBASE-5878:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12749586/HBASE-5878-v7-0.98.patch
  against 0.98 branch at commit c7065c4c40e94bcce2035b8ea9813cfc6124a7e0.
  ATTACHMENT ID: 12749586

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
22 warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn post-site goal 
to fail.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.regionserver.wal.TestHLog

 {color:red}-1 core zombie tests{color}.  There are 4 zombie test(s):   
at 
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testClassLoadingFromLibDirInJar(TestClassLoading.java:374)
at 
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testPreWALRestoreSkip(TestRegionObserverInterface.java:717)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//console

This message is automatically generated.

 Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
 ---

 Key: HBASE-5878
 URL: https://issues.apache.org/jira/browse/HBASE-5878
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch, 
 HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, 
 HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch, 
 HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch


 SequencFileLogReader: 
 Currently Hbase using getFileLength api from DFSInputStream class by 
 reflection. DFSInputStream is not exposed as public. So, this may change in 
 future. Now HDFS exposed HdfsDataInputStream as public API.
 We can make use of it, when we are not able to find the getFileLength api 
 from DFSInputStream as a else condition. So, that we will not have any sudden 
 surprise like we are facing today.
 Also,  it is just logging one warn message and proceeding if it throws any 
 exception while getting the length. I think we can re-throw the exception 
 because there is no point in continuing with dataloss.
 {code}
 long adjust = 0;
   try {
 Field fIn = FilterInputStream.class.getDeclaredField(in);
 fIn.setAccessible(true);
 Object realIn = fIn.get(this.in);
 // In hadoop 0.22, DFSInputStream is a standalone class.  Before 
 this,
 // it was an inner class of DFSClient.
 if (realIn.getClass().getName().endsWith(DFSInputStream)) {
   Method getFileLength = realIn.getClass().
 getDeclaredMethod(getFileLength, new Class? []{});
   getFileLength.setAccessible(true);
   long realLength =

[jira] [Commented] (HBASE-13376) Improvements to Stochastic load balancer

2015-08-10 Thread Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680257#comment-14680257
]

Ted Yu commented on HBASE-13376:

The change in BaseLoadBalancer.java was left out, leading to compilation error.

In your next patch, please add back the change. Also note recent refactoring in
TestStochasticLoadBalancer.java

Thanks

Improvements to Stochastic load balancer

Key: HBASE-13376
URL: https://issues.apache.org/jira/browse/HBASE-13376
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 1.0.0, 0.98.12
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Attachments: 13376-v2.txt, HBASE-13376.patch, HBASE-13376_0.98.txt,
HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt,
HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_98.patch,
HBASE-13376_branch-1.patch

There are two things this jira tries to address:
1. The locality picker in the stochastic balancer does not pick regions with
least locality as candidates for swap/move. So when any user configures
locality cost in the configs, the balancer does not always seems to move
regions with bad locality.
2. When a cluster has equal number of loaded regions, it always picks the
first one. It should pick a random region on one of the equally loaded
servers. This improves a chance of finding a good candidate, when load picker
is invoked several times.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13376) Improvements to Stochastic load balancer

2015-08-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-13376:
---
Status: Open  (was: Patch Available)

 Improvements to Stochastic load balancer
 

 Key: HBASE-13376
 URL: https://issues.apache.org/jira/browse/HBASE-13376
 Project: HBase
  Issue Type: Improvement
  Components: Balancer
Affects Versions: 0.98.12, 1.0.0
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
 Attachments: 13376-v2.txt, HBASE-13376.patch, HBASE-13376_0.98.txt, 
 HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt, 
 HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_98.patch, 
 HBASE-13376_branch-1.patch


 There are two things this jira tries to address:
 1. The locality picker in the stochastic balancer does not pick regions with 
 least locality as candidates for swap/move. So when any user configures 
 locality cost in the configs, the balancer does not always seems to move 
 regions with bad locality. 
 2. When a cluster has equal number of loaded regions, it always picks the 
 first one. It should pick a random region on one of the equally loaded 
 servers. This improves a chance of finding a good candidate, when load picker 
 is invoked several times. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14190) Assign system tables ahead of user region assignment

2015-08-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14190:
---
Attachment: (was: 14190-v4.txt)

 Assign system tables ahead of user region assignment
 

 Key: HBASE-14190
 URL: https://issues.apache.org/jira/browse/HBASE-14190
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Attachments: 14190-v5.txt


 Currently the namespace table region is assigned like user regions.
 I spent several hours working with a customer where master couldn't finish 
 initialization.
 Even though master was restarted quite a few times, it went down with the 
 following:
 {code}
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Master server abort: loaded coprocessors are: []
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Unhandled exception. Starting shutdown.
 java.io.IOException: Timedout 30ms waiting for namespace table to be 
 assigned
   at 
 org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
   at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
   at java.lang.Thread.run(Thread.java:744)
 {code}
 During previous run(s), namespace table was created, hence leaving an entry 
 in hbase:meta.
 The following if block in TableNamespaceManager#start() was skipped:
 {code}
 if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
   TableName.NAMESPACE_TABLE_NAME)) {
 {code}
 TableNamespaceManager#start() spins, waiting for namespace region to be 
 assigned.
 There was issue in master assigning user regions.
 We tried issuing 'assign' command from hbase shell which didn't work because 
 of the following check in MasterRpcServices#assignRegion():
 {code}
   master.checkInitialized();
 {code}
 This scenario can be avoided if we assign hbase:namespace table after 
 hbase:meta is assigned but before user table region assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

88 matches

Mail list logo