[jira] [Commented] (HBASE-21795) Client application may get stuck (time bound) if a table modify op is called immediately after split op

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755779#comment-16755779
 ] 

Bhupendra Kumar Jain commented on HBASE-21795:
--

I mean, clear parent region state from AssignmentMaanger's regionStates memory 
(Similar as Merge region procedure flow).. Not clearing from Meta table... 

Meta table clean up will be taken care by CatalogJanitor itself ... 

> Client application may get stuck (time bound) if a table modify op is called 
> immediately after split op
> ---
>
> Key: HBASE-21795
> URL: https://issues.apache.org/jira/browse/HBASE-21795
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Attachments: HBASE-21795.master.001.patch
>
>
> *Steps:*
>  * Create a table
>  * Split the table
>  * Modify the table immediately after splitting
> *Expected*: 
> The modify table procedure completes and control returns back to client
> *Actual:* 
> The modify table procedure completes and control does not return back to 
> client, until catalog janitor runs and deletes parent or future timeout occurs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21795) Client application may get stuck (time bound) if a table modify op is called immediately after split op

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755755#comment-16755755
 ] 

Bhupendra Kumar Jain commented on HBASE-21795:
--

{quote}Merge does not have this problem.{quote}
MergeTableRegionsProcedure clears the merged region entries from regionStates 
by invoking markRegionAsMerged() on MERGE_TABLE_REGIONS_UPDATE_META state but 
SplitTableRegionProcedure doesn't clears the parent region from regionStates 
and thus its counted in rit later 

{quote}Submitted a patch which reproduces the bug and proposes a fix.{quote}
Can we clear the parent region state during split ?

> Client application may get stuck (time bound) if a table modify op is called 
> immediately after split op
> ---
>
> Key: HBASE-21795
> URL: https://issues.apache.org/jira/browse/HBASE-21795
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Attachments: HBASE-21795.master.001.patch
>
>
> *Steps:*
>  * Create a table
>  * Split the table
>  * Modify the table immediately after splitting
> *Expected*: 
> The modify table procedure completes and control returns back to client
> *Actual:* 
> The modify table procedure completes and control does not return back to 
> client, until catalog janitor runs and deletes parent or future timeout occurs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708453#comment-16708453
 ] 

Bhupendra Kumar Jain edited comment on HBASE-17914 at 12/4/18 9:26 AM:
---

{quote}Which feature of Phoenix needs to hack StoreFileReader?
{quote}
As I know Phoenix Local Index feature uses this. Please refer class 
IndexHalfStoreFileReader in Phoenix code


was (Author: bhupendra):
{quote}Which feature of Phoenix needs to hack StoreFileReader?{quote}

As I know Phoenix Local Index feature uses this. Please refer class 
IndexHalfStoreFileReader





> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17914-v1.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v2.patch, HBASE-17914-v3.patch, HBASE-17914.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708453#comment-16708453
 ] 

Bhupendra Kumar Jain commented on HBASE-17914:
--

{quote}Which feature of Phoenix needs to hack StoreFileReader?{quote}

As I know Phoenix Local Index feature uses this. Please refer class 
IndexHalfStoreFileReader





> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17914-v1.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v2.patch, HBASE-17914-v3.patch, HBASE-17914.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21487) Concurrent modify table ops can lead to unexpected results

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708402#comment-16708402
 ] 

Bhupendra Kumar Jain commented on HBASE-21487:
--

{quote}record the seqenceID of the current tabledescriptor on FS(and pass to 
ModifyTableProcedure),
{quote}
I think , SequenceID concept is specific to FSTableDescriptors implementation 
which is one of the impl of TableDescriptors interface. So IMO we should not 
depend on sequenceID here for comparison.  

May be we can  pass the old_table_descriptor also in ModifyTableProcedure. In 
MODIFY_TABLE_PREPARE step, we can compare the old_table_descriptor with 
current_table_descriptor, if they are not same then we can throw exception ... 




> Concurrent modify table ops can lead to unexpected results
> --
>
> Key: HBASE-21487
> URL: https://issues.apache.org/jira/browse/HBASE-21487
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Syeda Arshiya Tabreen
>Priority: Major
>
> Concurrent  modifyTable or add/delete/modify columnFamily leads to incorrect 
> result. After HBASE-18893, The behavior of add/delete/modify column family 
> during concurrent operation is changed compare to branch-1.When  one client 
> is adding cf2 and another one cf3 .. In branch-1 final result will be 
> cf1,cf2,cf3 but now either cf1,cf2 OR cf1,cf3 will be the outcome depending 
> on which ModifyTableProcedure executed finally.Its because new table 
> descriptor is constructed before submitting the ModifyTableProcedure in 
> HMaster class and its not guarded by any lock.
> *Steps to reproduce*
> 1.Create table 't' with column family 'f1'
> 2.Client-1 and Client-2 requests to add column family 'f2' and 'f3' on table 
> 't' concurrently.
> *Expected Result*
> Table should have three column families(f1,f2,f3)
> *Actual Result*
> Table 't' will have column family either (f1,f2) or (f1,f3)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707104#comment-16707104
 ] 

Bhupendra Kumar Jain commented on HBASE-17914:
--

{quote}but could you please find another way to implement the feature?{quote}
Couldn't think of any clean approach .. Please suggest if you have any better 
way for this API
{quote}For now imaybe it is fine to add these parameters as this method is only 
suppose to be used by Phoenix {quote}
Better to add these parameters as of now.  Will open another JIRA for this ...

[~chrajeshbab...@gmail.com]  [~anoop.hbase]  Phoenix LocalIndex may also have 
this issue ?

> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17914-v1.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v2.patch, HBASE-17914-v3.patch, HBASE-17914.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703024#comment-16703024
 ] 

Bhupendra Kumar Jain commented on HBASE-17914:
--

Ping [~chrajeshbab...@gmail.com], [~anoop.hbase]

> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17914-v1.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v2.patch, HBASE-17914-v3.patch, HBASE-17914.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701850#comment-16701850
 ] 

Bhupendra Kumar Jain commented on HBASE-17914:
--

In {{StoreFileReader}} class , a new parameter {{shared}} is introduced to 
control stream read / pread. But this parameter is not passed to the 
co-processor in {{preStoreFileReaderOpen()}}
So co-processor when creating its own StoreFileReader, has not awareness about 
type of read. Further based on the parameter value, the readCompleted() method 
in StoreFileReader will close the actual HFile reader and underlying streams.

In one of our scenario, with co-processor returning its own StoreFileReader 
[shared=false], the scan successfully happens only first time. Next scan 
operation are resulting in stream closed exception. (I think, this case may 
happens in phoenix IndexHalfStoreFileReader also if HBase compaction is 
disabled)

So I think , the new parameter {{shared}} and {{refCount}} should be available 
to the co-processor

> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17914-v1.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v2.patch, HBASE-17914-v3.patch, HBASE-17914.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21500) Jetty aliases parameter need to change as per jetty 9.x version

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-21500:
-
Description: 
Noticed that Jetty aliases parameter in HttpServer.java  
"*org.mortbay.jetty.servlet.Default.aliases*" is as per old jetty version and  
need to change as per jetty 9.x new version after the HBASE-12894

Refer 
https://github.com/apache/hbase/blob/405bf5e6383a09f435baadbac6c389e9f6c43ac6/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L647

It should be *"org.eclipse.jetty.servlet.Default.aliases"* 

  was:
Noticed that Jetty aliases parameter 
"*org.mortbay.jetty.servlet.Default.aliases*" is as per old jetty version and  
need to change as per jetty 9.x new version after the HBASE-12894

It should be *"org.eclipse.jetty.servlet.Default.aliases"* 


> Jetty aliases parameter need to change as per jetty 9.x version
> ---
>
> Key: HBASE-21500
> URL: https://issues.apache.org/jira/browse/HBASE-21500
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0, 2.0.1, 2.1.1
>Reporter: Bhupendra Kumar Jain
>Priority: Minor
>
> Noticed that Jetty aliases parameter in HttpServer.java  
> "*org.mortbay.jetty.servlet.Default.aliases*" is as per old jetty version and 
>  need to change as per jetty 9.x new version after the HBASE-12894
> Refer 
> https://github.com/apache/hbase/blob/405bf5e6383a09f435baadbac6c389e9f6c43ac6/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L647
> It should be *"org.eclipse.jetty.servlet.Default.aliases"* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21500) Jetty aliases parameter need to change as per jetty 9.x version

[Bhupendra Kumar Jain (JIRA)]

Bhupendra Kumar Jain created HBASE-21500:


 Summary: Jetty aliases parameter need to change as per jetty 9.x 
version
 Key: HBASE-21500
 URL: https://issues.apache.org/jira/browse/HBASE-21500
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.1.1, 2.0.1, 2.0.0, 2.1.0, 3.0.0
Reporter: Bhupendra Kumar Jain


Noticed that Jetty aliases parameter 
"*org.mortbay.jetty.servlet.Default.aliases*" is as per old jetty version and  
need to change as per jetty 9.x new version after the HBASE-12894

It should be *"org.eclipse.jetty.servlet.Default.aliases"* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-18198) Node creation fails with NPE if ACLs are null

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain resolved HBASE-18198.
--
Resolution: Invalid

Raised in wrong component. 

> Node creation fails with NPE if ACLs are null
> -
>
> Key: HBASE-18198
> URL: https://issues.apache.org/jira/browse/HBASE-18198
> Project: HBase
>  Issue Type: Bug
>Reporter: Bhupendra Kumar Jain
>
> If null ACLs are passed then zk node creation fails with NPE
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
> {code}
> Below APIs have problem.
> {code}
> public void create(final String path, byte data[], List acl,
> CreateMode createMode, StringCallback cb, Object ctx)
> public void create(final String path, byte data[], List acl,
> CreateMode createMode, Create2Callback cb, Object ctx)
> {code}
> Solution:  Need to handle NULL acl in removeDuplicates method in server.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-18198) Node creation fails with NPE if ACLs are null

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-18198:
-
Description: 
If null ACLs are passed then zk node creation fails with NPE
{code}
java.lang.NullPointerException
at 
org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
at 
org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
{code}

Below APIs have problem.
{code}
public void create(final String path, byte data[], List acl,
CreateMode createMode, StringCallback cb, Object ctx)

public void create(final String path, byte data[], List acl,
CreateMode createMode, Create2Callback cb, Object ctx)
{code}

Solution:  Need to handle NULL acl in removeDuplicates method in server.

  was:
If null ACLs are passed then zk node creation fails with NPE
{code}
java.lang.NullPointerException
at 
org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
at 
org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
{code}

Below APIs have problem.
{code}
public void create(final String path, byte data[], List acl,
CreateMode createMode, StringCallback cb, Object ctx)

public void create(final String path, byte data[], List acl,
CreateMode createMode, Create2Callback cb, Object ctx)
{code}


> Node creation fails with NPE if ACLs are null
> -
>
> Key: HBASE-18198
> URL: https://issues.apache.org/jira/browse/HBASE-18198
> Project: HBase
>  Issue Type: Bug
>Reporter: Bhupendra Kumar Jain
>
> If null ACLs are passed then zk node creation fails with NPE
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
> {code}
> Below APIs have problem.
> {code}
> public void create(final String path, byte data[], List acl,
> CreateMode createMode, StringCallback cb, Object ctx)
> public void create(final String path, byte data[], List acl,
> CreateMode createMode, Create2Callback cb, Object ctx)
> {code}
> Solution:  Need to handle NULL acl in removeDuplicates method in server.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-18198) Node creation fails with NPE if ACLs are null

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-18198:
-
Description: 
If null ACLs are passed then zk node creation fails with NPE
{code}
java.lang.NullPointerException
at 
org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
at 
org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
{code}

Below APIs have problem.
{code}
public void create(final String path, byte data[], List acl,
CreateMode createMode, StringCallback cb, Object ctx)

public void create(final String path, byte data[], List acl,
CreateMode createMode, Create2Callback cb, Object ctx)
{code}

  was:
If null ACLs are passed then zk node creation fails with NPE
{code}
java.lang.NullPointerException
at 
org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
at 
org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
{code}

Below APIs have problem.
{code}
public void create(final String path, byte data[], List acl,
CreateMode createMode, StringCallback cb, Object ctx)

public void create(final String path, byte data[], List acl,
CreateMode createMode, Create2Callback cb, Object ctx)


> Node creation fails with NPE if ACLs are null
> -
>
> Key: HBASE-18198
> URL: https://issues.apache.org/jira/browse/HBASE-18198
> Project: HBase
>  Issue Type: Bug
>Reporter: Bhupendra Kumar Jain
>
> If null ACLs are passed then zk node creation fails with NPE
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
>   at 
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
> {code}
> Below APIs have problem.
> {code}
> public void create(final String path, byte data[], List acl,
> CreateMode createMode, StringCallback cb, Object ctx)
> public void create(final String path, byte data[], List acl,
> CreateMode createMode, Create2Callback cb, Object ctx)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-18198) Node creation fails with NPE if ACLs are null

[Bhupendra Kumar Jain (JIRA)]

Bhupendra Kumar Jain created HBASE-18198:


 Summary: Node creation fails with NPE if ACLs are null
 Key: HBASE-18198
 URL: https://issues.apache.org/jira/browse/HBASE-18198
 Project: HBase
  Issue Type: Bug
Reporter: Bhupendra Kumar Jain


If null ACLs are passed then zk node creation fails with NPE
{code}
java.lang.NullPointerException
at 
org.apache.zookeeper.server.PrepRequestProcessor.removeDuplicates(PrepRequestProcessor.java:1301)
at 
org.apache.zookeeper.server.PrepRequestProcessor.fixupACL(PrepRequestProcessor.java:1341)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequestProcessor.java:519)
at 
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:1126)
at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:178)
{code}

Below APIs have problem.
{code}
public void create(final String path, byte data[], List acl,
CreateMode createMode, StringCallback cb, Object ctx)

public void create(final String path, byte data[], List acl,
CreateMode createMode, Create2Callback cb, Object ctx)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17779) disable_table_replication returns misleading message and does not turn off repliaton

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924096#comment-15924096
 ] 

Bhupendra Kumar Jain commented on HBASE-17779:
--

V2 patch looks fine to me. I think , we can include this in all branches as 
there is no incompatibility introduced for API and behavior change is actually 
a bug fix.  Whats your opinion [~tedyu]

> disable_table_replication returns misleading message and does not turn off 
> repliaton
> 
>
> Key: HBASE-17779
> URL: https://issues.apache.org/jira/browse/HBASE-17779
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Janos Gub
>  Labels: incompatibleChange
> Attachments: HBASE-17779.patch, HBASE-17779-v2.patch
>
>
> Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns 
> families is not replicated.
> Because of this if you have a table where replication is partially enabled, 
> calling disable_table_replication will not have any effect, but will report 
> that replication for a given table is disabled.
> Workaround is enabling table replication before disabling. 
> As a solution quoted from HBASE-17460:
> Admin#disableTableReplication() returns nothing and isTableRepEnabled() 
> returns boolean. For master branch, we can let isTableRepEnabled() return 
> enum (partially disabled, disabled, etc) This way, 
> Admin#disableTableReplication() can return meaningful value to the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17779) disable_table_replication returns misleading message and does not turn off repliaton

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923842#comment-15923842
 ] 

Bhupendra Kumar Jain commented on HBASE-17779:
--

Patch looks fine. Just few comments
1. Please change existing parameter name from "isRepEnabled" to "enableRep"  ( 
I think that is more meaningful )
2. Assign the result of isTableRepEnabled(htd) to a variable and use in if 
statement. ( To avoid calling isTableRepEnabled twice)

bq. In this one enable_table_replication and disable_table_replication always 
sets the replication scope for a given table, so there is no need to propagate 
other info to the user. 
Yes, No need to propagate other info the user. We can probably add separate 
public API for user to know the replication state. 
bq. Also it is a behavior change.
The patch corrects the current behavior which IMO was a bug.


> disable_table_replication returns misleading message and does not turn off 
> repliaton
> 
>
> Key: HBASE-17779
> URL: https://issues.apache.org/jira/browse/HBASE-17779
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Janos Gub
> Attachments: HBASE-17779.patch
>
>
> Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns 
> families is not replicated.
> Because of this if you have a table where replication is partially enabled, 
> calling disable_table_replication will not have any effect, but will report 
> that replication for a given table is disabled.
> Workaround is enabling table replication before disabling. 
> As a solution quoted from HBASE-17460:
> Admin#disableTableReplication() returns nothing and isTableRepEnabled() 
> returns boolean. For master branch, we can let isTableRepEnabled() return 
> enum (partially disabled, disabled, etc) This way, 
> Admin#disableTableReplication() can return meaningful value to the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907310#comment-15907310
 ] 

Bhupendra Kumar Jain commented on HBASE-17460:
--

bq. Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the 
columns families is not replicated, meaning that if you have a table with even 
one column family without replication you could not disable replication.
I think , This is a bug and we should fix it.
bq. Admin#disableTableReplication() returns nothing and isTableRepEnabled() 
returns boolean. For master branch, we can let isTableRepEnabled() return enum 
(partially disabled, disabled, etc) This way, Admin#disableTableReplication() 
can return meaningful value to the user.
Agree. We can enhance this in master branch

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>  Labels: replication
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, 
> 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, 
> HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, 
> HBASE-17460-addendum2-v4.patch, 
> HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, 
> HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, 
> HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, 
> HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902525#comment-15902525
 ] 

Bhupendra Kumar Jain commented on HBASE-17460:
--

Release notes can be removed as there is no behavior change introduced now.  
Thanks for the patch. 

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>  Labels: replication
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, 
> 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, 
> HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, 
> HBASE-17460-addendum2-v4.patch, 
> HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, 
> HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, 
> HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, 
> HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894396#comment-15894396
 ] 

Bhupendra Kumar Jain commented on HBASE-17460:
--

Addendum looks fine to me. I think, You also need to remove 
isReplicationEnabled() method as it will become unused method after this 
addendum . 

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>  Labels: incompatibleChange, replication
> Fix For: 2.0.0
>
> Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, 
> 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, 
> HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, 
> HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after region close

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867746#comment-15867746
 ] 

Bhupendra Kumar Jain commented on HBASE-17374:
--

As I went thru the JIRA and patch 
Jira desc says "The instance of TableAuthManager should not be close 
unless regionserver is closing."

   1) If Accesscontroller coprocessor is configured for regionserver, then 
there will be always one TableAuthManager instance. So the issue should not 
happen
   2) If Accesscontroller coprocessor is not configured for regionserver, then 
the issue will happen. This patch only handles the exception case which occurs 
during zk event processing and executor is already shutdown. 

But I think , as zk watcher holds the ZKPermissionWatcher listener instance, so 
ZKPermissionWatcher will keep getting the events (even though already closed) 
and it will lead to memory leak of TableAuthManager instance also.

So to solve this completely, ZKPermissionWatcher should unregister itself from 
the watcher in close() method. with that it will not get any events further and 
it will avoid memory leak.

Please correct if any other understanding. 

> ZKPermissionWatcher crashed when grant after region close
> -
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Assignee: Liu Junhong
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 0001-fix-for-HBASE-17374-20161228.patch, 
> 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> 

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865086#comment-15865086
 ] 

Bhupendra Kumar Jain commented on HBASE-17460:
--

This patch will have one incompatible behavior change for enable table 
replication API. 
As per the earlier behavior, If the replication is enabled , user executes the 
enable table replication command again and If table descriptor matches , It 
will check the replication scope for each column family and if any of the 
column family scope is not enabled, it will enable it. otherwise it will do 
nothing and final result is success. But as per the patch, it will throw 
exception if replication is already enabled.

I think, To handle the cyclic replication case, Only ignoring the replication 
scope from comparison should be enough. No need to check to 
isReplicationEnabled for local cluster. 

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>  Labels: replication
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17460.branch-1.v3.txt, HBASE-17460.patch, 
> HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17635) enable_table_replication script cannot handle replication scope

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863476#comment-15863476
 ] 

Bhupendra Kumar Jain commented on HBASE-17635:
--

HBASE-17460  skips the replication scope attribute comparison while comparing 
descriptors. I think that should resolve this.

> enable_table_replication script cannot handle replication scope
> ---
>
> Key: HBASE-17635
> URL: https://issues.apache.org/jira/browse/HBASE-17635
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.3.1
>Reporter: Lars George
>
> When you add a peer, then enable a table for replication using 
> {{enable_table_replication}}, the script will create the table on the peer 
> cluster, but with one difference:
> _Master Cluster_:
> {noformat}
> hbase(main):027:0> describe 'testtable'
> Table testtable is ENABLED
>   
> 
> testtable 
>   
> 
> COLUMN FAMILIES DESCRIPTION   
>   
> 
> {NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', 
> REPLICATION_SCOPE => '1', VERSIONS => '1', COMPRESSION => 'NONE', 
> MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE',
>  BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>   
> 
> 1 row(s) in 0.0700 seconds
> {noformat}
> _Peer Cluster_:
> {noformat}
> hbase(main):003:0> describe 'testtable'
> Table testtable is ENABLED
>   
> 
> testtable 
>   
> 
> COLUMN FAMILIES DESCRIPTION   
>   
> 
> {NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', 
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 
> 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE',
>  BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>   
> 
> 1 row(s) in 0.1260 seconds
> {noformat}
> Note that the replication scope is different. Removing the peer, adding it 
> again and enabling the table gives this now:
> {noformat}
> hbase(main):026:0> enable_table_replication 'testtable'
> ERROR: Table testtable exists in peer cluster 1, but the table descriptors 
> are not same when compared with source cluster. Thus can not enable the 
> table's replication switch.
> {noformat}
> That is dumb, as it was the same script that enabled the replication scope in 
> the first place. It should skip that particular attribute when comparing the 
> cluster schemas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16144) Replication queue's lock will live forever if RS acquiring the lock has died prematurely

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359042#comment-15359042
 ] 

Bhupendra Kumar Jain commented on HBASE-16144:
--

Thanks for clarification. 

> Replication queue's lock will live forever if RS acquiring the lock has died 
> prematurely
> 
>
> Key: HBASE-16144
> URL: https://issues.apache.org/jira/browse/HBASE-16144
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16144-v1.patch, HBASE-16144-v2.patch
>
>
> In default, we will use multi operation when we claimQueues from ZK. But if 
> we set hbase.zookeeper.useMulti=false, we will add a lock first, then copy 
> nodes, finally clean old queue and the lock. 
> However, if the RS acquiring the lock crash before claimQueues done, the lock 
> will always be there and other RS can never claim the queue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16144) Replication queue's lock will live forever if RS acquiring the lock has died prematurely

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358567#comment-15358567
 ] 

Bhupendra Kumar Jain commented on HBASE-16144:
--

If the lock is ephemeral node in zk, then on RS crash, it will be deleted 
automatically. Any issues by making lock as ephemeral node ? 
there are some discussion in HBASE-12241 and HBASE-2611 also. 

> Replication queue's lock will live forever if RS acquiring the lock has died 
> prematurely
> 
>
> Key: HBASE-16144
> URL: https://issues.apache.org/jira/browse/HBASE-16144
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16144-v1.patch, HBASE-16144-v2.patch
>
>
> In default, we will use multi operation when we claimQueues from ZK. But if 
> we set hbase.zookeeper.useMulti=false, we will add a lock first, then copy 
> nodes, finally clean old queue and the lock. 
> However, if the RS acquiring the lock crash before claimQueues done, the lock 
> will always be there and other RS can never claim the queue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010766#comment-15010766
 ] 

Bhupendra Kumar Jain commented on HBASE-14777:
--

Thanks Ashu and Appy
I was on festival vacation, So didn't check this further. 

V3 looks good. Can you also add the failure scenario in test 
case(DummyReplicator can throw exception). This can cover the scenarios when 
entry from entryLists should not be removed and retried. 

Initially I observed the index problem during my code review and during 
simulation of the same, I was printing the index returned from the future like 
below code
{code}
int i = (int)f.get();
entryLists.remove(i);
{code}
Since I was assigning as int , so thats the reason, I got 
IndexOutOfBoundsException. But as [~ashu210890] pointed out, before index 
problem, the other problem of removal exists. 

Thanks for looking into this

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14777-1.patch, HBASE-14777-2.patch, 
> HBASE-14777-3.patch, HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14777:
-
Assignee: Ashu Pachauri  (was: Bhupendra Kumar Jain)

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Ashu Pachauri
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14777-1.patch, HBASE-14777-2.patch, 
> HBASE-14777-3.patch, HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996498#comment-14996498
 ] 

Bhupendra Kumar Jain commented on HBASE-14777:
--

Thanks all for the review. I will try to add the test case for this scenario. 

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
> Attachments: HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Bhupendra Kumar Jain (JIRA)]

Bhupendra Kumar Jain created HBASE-14777:


 Summary: Replication fails with IndexOutOfBoundsException
 Key: HBASE-14777
 URL: https://issues.apache.org/jira/browse/HBASE-14777
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 2.0.0, 1.2.0, 1.3.0
Reporter: Bhupendra Kumar Jain
Assignee: Bhupendra Kumar Jain
Priority: Critical


Replication fails with IndexOutOfBoundsException 
{code}
regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
 threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(Unknown Source)
at java.util.ArrayList.remove(Unknown Source)
at 
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
{code}

Its happening due to incorrect removal of entries from the replication entries 
list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993479#comment-14993479
 ] 

Bhupendra Kumar Jain commented on HBASE-14777:
--

The code removes the successful entries from the list of entries. Each removal 
from the list changes the position of subsequent element of list which results 
in IndexOutOfBoundsException
{code}
for (Future f : futures) {
  try {
// wait for all futures, remove successful parts
// (only the remaining parts will be retried)
entryLists.remove(f.get());
  } catch (InterruptedException ie) {
iox =  new IOException(ie);
  }
{code}

To handle this, We can iterate and remove in reverse order. 
{code}
 int fLen = futures.size();
for (int fIndex = fLen - 1; fIndex >= 0; fIndex--) {
  try {
// wait for all futures, remove successful parts
// (only the remaining parts will be retried)
entryLists.remove(futures.get(fIndex).get());
} catch (InterruptedException ie) {
iox =  new IOException(ie);
  }

{code}


> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14777:
-
Status: Patch Available  (was: Open)

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
> Attachments: HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14777:
-
Attachment: HBASE-14777.patch

Please review the attached patch

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
> Attachments: HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964954#comment-14964954
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

Test case failures are not related to this patch.

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.1_v1.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.2_v1.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: HBASE-14366-branch-1.1_v1.patch

Corrected check Style error. Please check

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.1_v1.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.2_v1.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: HBASE-14366-branch-1.2_v1.patch

Corrected check Style error. Please check

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.2.patch, HBASE-14366-branch-1.2_v1.patch, 
> HBASE-14366-branch-1.patch, HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: HBASE-14366-branch-1.1.patch

Patch for branch 1.1. It can also apply on branch 1.0.
Please check.

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.2.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: HBASE-14366-branch-1.2.patch

Patch for branch 1.2

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.patch, HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13982) Add info for visibility labels/cell TTLs to ImportTsv

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950247#comment-14950247
 ] 

Bhupendra Kumar Jain commented on HBASE-13982:
--

Before adding usage, I was verifying these features.   Its fine, you can take 
this up. 

> Add info for visibility labels/cell TTLs to ImportTsv
> -
>
> Key: HBASE-13982
> URL: https://issues.apache.org/jira/browse/HBASE-13982
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.1.0.1
>Reporter: Lars George
>Assignee: Bhupendra Kumar Jain
>  Labels: beginner
> Fix For: 2.0.0, 1.3.0
>
>
> HBASE-9832 added support for two more optional, special TSV columns, but no 
> usage info was added. Add.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14520) Optimize the number of calls for tags creation in bulk load

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946289#comment-14946289
 ] 

Bhupendra Kumar Jain commented on HBASE-14520:
--

Thanks Ted and Anoop for review and commit. 

> Optimize the number of calls for tags creation in bulk load
> ---
>
> Key: HBASE-14520
> URL: https://issues.apache.org/jira/browse/HBASE-14520
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0
>
> Attachments: HBASE-14520.patch
>
>
> At present, ttl and Visibility expr is one per tsv line i.e. the values and 
> the tags remain same for all the columns present in that line. As per the 
> code, List of tags are created for each cell, Instead of creating new tags 
> for each cell, tags created once for the line can be reused by other cells.  
> Assume 1Million rows and 1000 columns. Currently tags creation will happen 
> for 1M * 1000 times. If reuse the tags, the tags creation can reduce to 1M 
> times. (i.e. one per tsv line). 
> This is applicable in both TsvImporterMapper and TextSortReducer logic. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946287#comment-14946287
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

javadoc warnings are not related to the patch.

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.patch, HBASE-14366_2(1).patch, 
> HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: HBASE-14366-0.98.patch
HBASE-14366-branch-1.patch

Attached patches for 0.98 and branch-1. Please review 

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.patch, HBASE-14366_2(1).patch, 
> HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14520) Optimize the number of calls for tags creation in bulk load

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14520:
-
Fix Version/s: 2.0.0
   Status: Patch Available  (was: Open)

> Optimize the number of calls for tags creation in bulk load
> ---
>
> Key: HBASE-14520
> URL: https://issues.apache.org/jira/browse/HBASE-14520
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0
>
> Attachments: HBASE-14520.patch
>
>
> At present, ttl and Visibility expr is one per tsv line i.e. the values and 
> the tags remain same for all the columns present in that line. As per the 
> code, List of tags are created for each cell, Instead of creating new tags 
> for each cell, tags created once for the line can be reused by other cells.  
> Assume 1Million rows and 1000 columns. Currently tags creation will happen 
> for 1M * 1000 times. If reuse the tags, the tags creation can reduce to 1M 
> times. (i.e. one per tsv line). 
> This is applicable in both TsvImporterMapper and TextSortReducer logic. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14520) Optimize the number of calls for tags creation in bulk load

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14520:
-
Summary: Optimize the number of calls for tags creation in bulk load  (was: 
Optimnize the number of calls for tags creation in bulk load)

> Optimize the number of calls for tags creation in bulk load
> ---
>
> Key: HBASE-14520
> URL: https://issues.apache.org/jira/browse/HBASE-14520
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>
> At present, ttl and Visibility expr is one per tsv line i.e. the values and 
> the tags remain same for all the columns present in that line. As per the 
> code, List of tags are created for each cell, Instead of creating new tags 
> for each cell, tags created once for the line can be reused by other cells.  
> Assume 1Million rows and 1000 columns. Currently tags creation will happen 
> for 1M * 1000 times. If reuse the tags, the tags creation can reduce to 1M 
> times. (i.e. one per tsv line). 
> This is applicable in both TsvImporterMapper and TextSortReducer logic. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943306#comment-14943306
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

In the test cases there are 2 data lines (one with valid vis expression and 
another with invalid vis expression) each having 2 columns.  After the bulk 
load happens, the invalid line will be skipped as bad line. Test case compares 
the actual KV count written in HFile with expected KV count to ensure that 
invalid vis expression KVs are skipped 
{code}
assertTrue(String.format(
"KV count in output hfile=<%d> doesn't match with expected KV 
count=<%d>", actualKVCount,
expectedKVCount), actualKVCount == expectedKVCount);
{code}

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14520) Optimize the number of calls for tags creation in bulk load

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14520:
-
Attachment: HBASE-14520.patch

Please review the patch. 

Tags are created for each tsv line once and reused for all the columns present 
in that line.

> Optimize the number of calls for tags creation in bulk load
> ---
>
> Key: HBASE-14520
> URL: https://issues.apache.org/jira/browse/HBASE-14520
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Attachments: HBASE-14520.patch
>
>
> At present, ttl and Visibility expr is one per tsv line i.e. the values and 
> the tags remain same for all the columns present in that line. As per the 
> code, List of tags are created for each cell, Instead of creating new tags 
> for each cell, tags created once for the line can be reused by other cells.  
> Assume 1Million rows and 1000 columns. Currently tags creation will happen 
> for 1M * 1000 times. If reuse the tags, the tags creation can reduce to 1M 
> times. (i.e. one per tsv line). 
> This is applicable in both TsvImporterMapper and TextSortReducer logic. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14520) Optimnize the number of calls for tags creation in bulk load

[Bhupendra Kumar Jain (JIRA)]

Bhupendra Kumar Jain created HBASE-14520:


 Summary: Optimnize the number of calls for tags creation in bulk 
load
 Key: HBASE-14520
 URL: https://issues.apache.org/jira/browse/HBASE-14520
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Bhupendra Kumar Jain
Assignee: Bhupendra Kumar Jain


At present, ttl and Visibility expr is one per tsv line i.e. the values and the 
tags remain same for all the columns present in that line. As per the code, 
List of tags are created for each cell, Instead of creating new tags for each 
cell, tags created once for the line can be reused by other cells.  

Assume 1Million rows and 1000 columns. Currently tags creation will happen for 
1M * 1000 times. If reuse the tags, the tags creation can reduce to 1M times. 
(i.e. one per tsv line). 

This is applicable in both TsvImporterMapper and TextSortReducer logic. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable.

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936474#comment-14936474
 ] 

Bhupendra Kumar Jain commented on HBASE-14498:
--

To fix this one of the approach can be as below: 

Start a timer thread on receiving the Disconnected event. The timer thread will 
check after every 100 ms  if any SyncConnected event is received.

if SyncConnected event is received, then it means, the ZK connection is 
established again. So this thread can stop and no need of any further action. 

if SyncConnected is not received in specified time interval (t time) , then 
this thread can initiate master abort. 

The time interval (t) should be less than the ZK Session time out. (May be 
2/3rd of session time out value ) , This is to make sure that standby HM will 
not become active within this time period. 

> Master stuck in infinite loop when all Zookeeper servers are unreachable.
> -
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 1.0.1
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to 
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)] 
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in 
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and 
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)] 
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)] 
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Opening socket connection to server 
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown 
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Opening socket connection to server 
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown 
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in 
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and 
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000] 
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, 
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181, 
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2015-09-26 15:25:17,872 INFO [main-SendThread(ZK-Host:2181)] 
> client.FourLetterWordMain: connecting to ZK-Host 2181
> 2015-09-26 15:25:19,874 WARN [main-SendThread(ZK-Host:2181)] 
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host
> 2015-09-26 15:25:19,874 INFO [main-SendThread(ZK-Host:2181)] 
> zookeeper.ClientCnxn: Opening socket connection to server ZK-Host/ZK-IP:2181. 
> Will not attempt to authenticate using SASL (unknown error)
> {code}
> > Since HM1 was not able to connect to any ZK, so session timeout didnt 
> > happen at Zookeeper server side and HM1 didnt abort.
> > On Zookeeper session timeout standby master (HM2) registered himself as an 
> > active master. 
> > HM2 is keep on waiting for region server to report him as part of active 
> > master intialization.
> {noformat} 
> 2015-09-26 15:24:44,928 | INFO | HM2-Host:21300.activeMasterManager | Waiting 
> for region servers count to settle; currently checked in 0, slept for 0 ms, 
> expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval 
> of 1500 ms. | 
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> ---
> ---
> 2015-09-26 15:32:50,841 | INFO | HM2-Host:21300.activeMasterManager | Waiting 
> for region servers count to settle; currently checked in 0, slept for 483913 
> ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, 
> interval of 1500 ms. | 
> 

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933124#comment-14933124
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

In case the file is invalid (i.e. fail to parse), then it will be considered as 
BadLine scenario in importTsv. 
Any other scenario you are referring to ?

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14365) Error scanning 'labels' table in logs with exception while running bulkload, even visibility feature is disabled

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908056#comment-14908056
 ] 

Bhupendra Kumar Jain commented on HBASE-14365:
--

Yes [~anoopsamjohn], I will add the test cases to assert all these cases. To 
avoid conflict I will submit the new patch after [HBASE-14366] gets committed. 

> Error scanning 'labels' table in logs with exception while running bulkload, 
> even visibility feature is disabled
> 
>
> Key: HBASE-14365
> URL: https://issues.apache.org/jira/browse/HBASE-14365
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14365.patch
>
>
> If visibility feature is disabled, Below exception in logs during importtsv 
> run. In case feature is disabled, No need to log the below as error. Its is 
> bit misleading for the user.
> {code}
> ERROR [main] 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver: Error 
> scanning 'labels' table
> org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:858)
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:756)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:1)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:184)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:311)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:286)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:162)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at 
> org.apache.hadoop.hbase.client.ClientSimpleScanner.(ClientSimpleScanner.java:42)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:381)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: HBASE-14366_2(1).patch

All importtsv related test cases got passed, but Some test cases (not related 
to this ) got skipped in last QA run. So retrying the same patch for another 
Hadoop QA run. 


> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: HBASE-14366_2.patch

Attaching a new patch which addresses the review comments and includes test 
cases for invalid visibility label scenario. Please review. 

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747136#comment-14747136
 ] 

Bhupendra Kumar Jain commented on HBASE-14380:
--

Thanks Ted for committing the patches. 

> Correct data gets skipped along with bad data in importTsv bulk load thru 
> TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3
>
> Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, 
> HBASE-14380-branch-1.2-v1.patch, HBASE-14380-branch-1.2.patch, 
> HBASE-14380_v1.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14380:
-
Attachment: HBASE-14380-branch-1.2-v1.patch

> Correct data gets skipped along with bad data in importTsv bulk load thru 
> TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, 
> HBASE-14380-branch-1.2-v1.patch, HBASE-14380-branch-1.2.patch, 
> HBASE-14380_v1.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745328#comment-14745328
 ] 

Bhupendra Kumar Jain commented on HBASE-14380:
--

My bad, test case failure is due to parameter valueMultiplier not passed 
properly in overloaded method call. 
Attached v1 patch for branch 1.2 .. Please check.  

The master branch patch doesn't have the test case failures. 

> Correct data gets skipped along with bad data in importTsv bulk load thru 
> TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, 
> HBASE-14380-branch-1.2-v1.patch, HBASE-14380-branch-1.2.patch, 
> HBASE-14380_v1.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14380:
-
Status: Patch Available  (was: Reopened)

> Correct data gets skipped along with bad data in importTsv bulk load thru 
> TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, 
> HBASE-14380-branch-1.2-v1.patch, HBASE-14380-branch-1.2.patch, 
> HBASE-14380_v1.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13153) enable bulkload to support replication

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745466#comment-14745466
 ] 

Bhupendra Kumar Jain commented on HBASE-13153:
--

Thanks all for reviewing the design. One small update.

As per earlier design, we are keeping references of hfiles into ZK to avoid 
hfile clean up before replication is finished. There is small change require in 
design to handle multiple peer cluster replication scenario. These hfile 
references needs to be maintained in zk for each peer based on replication 
scope enabled for that peer. 

Once the replication of hfile for any of the peer is successfully done, 
corresponding zk reference entry will be cleared. We will also update the 
design doc with this change. 

> enable bulkload to support replication
> --
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load 
> Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14380:
-
Attachment: HBASE-14380-branch-1.2.patch

> Correct data gets skipped along with bad data in importTsv bulk load thru 
> TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, 
> HBASE-14380-branch-1.2.patch, HBASE-14380_v1.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743294#comment-14743294
 ] 

Bhupendra Kumar Jain commented on HBASE-14380:
--

Thanks Ted for committing the previous patch. I have attached the patch for 
branch 1.2 also. Please review.

> Correct data gets skipped along with bad data in importTsv bulk load thru 
> TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, 
> HBASE-14380-branch-1.2.patch, HBASE-14380_v1.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14365) Error scanning 'labels' table in logs with exception while running bulkload, even visibility feature is disabled

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743304#comment-14743304
 ] 

Bhupendra Kumar Jain commented on HBASE-14365:
--

If visibility label feature is not enable but data contains the visibility 
expression, then during bulk load the expression resolution will be failed and 
that data will not be loaded. So there will be no security issue. 

After committing [HBASE-14366] this scenario will be counted as bad line too. 

> Error scanning 'labels' table in logs with exception while running bulkload, 
> even visibility feature is disabled
> 
>
> Key: HBASE-14365
> URL: https://issues.apache.org/jira/browse/HBASE-14365
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14365.patch
>
>
> If visibility feature is disabled, Below exception in logs during importtsv 
> run. In case feature is disabled, No need to log the below as error. Its is 
> bit misleading for the user.
> {code}
> ERROR [main] 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver: Error 
> scanning 'labels' table
> org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:858)
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:756)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:1)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:184)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:311)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:286)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:162)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at 
> org.apache.hadoop.hbase.client.ClientSimpleScanner.(ClientSimpleScanner.java:42)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:381)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740757#comment-14740757
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

Thanks [~anoop.hbase] for comments. To avoid conflicts, I will upload the new 
patch after [HBASE-14380] gets committed. 

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14380:
-
Attachment: HBASE-14380_v1.patch

Attached v1 patch for review. Patch includes the test case for invalid data 
scenario. Please check.

> Correct data also getting skipped along with bad data in importTsv bulk load 
> thru TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Attachments: 0001-HBASE-14380.patch, HBASE-14380_v1.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain reassigned HBASE-14380:


Assignee: Bhupendra Kumar Jain  (was: Bhupendra)

> Correct data also getting skipped along with bad data in importTsv bulk load 
> thru TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

Bhupendra Kumar Jain created HBASE-14380:


 Summary: Correct data also getting skipped along with bad data in 
importTsv bulk load thru TsvImporterTextMapper
 Key: HBASE-14380
 URL: https://issues.apache.org/jira/browse/HBASE-14380
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bhupendra Kumar Jain
Assignee: Bhupendra


Cosider the input data is as below 
ROWKEY, TIEMSTAMP, Col_Value
r1,1,v1 >> Correct line
r1   >> Bad line
r1,3,v3 >> Correct line
r1,4,v4 >> Correct line

When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper , 
 All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14380:
-
Attachment: 0001-HBASE-14380.patch

Simple patch. Please review.

> Correct data also getting skipped along with bad data in importTsv bulk load 
> thru TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Attachments: 0001-HBASE-14380.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734734#comment-14734734
 ] 

Bhupendra Kumar Jain commented on HBASE-14380:
--

TextSortReducer will receive the request grouped by rowkey and all text lines 
as Iterable values. 
{code}
protected void reduce(ImmutableBytesWritable rowKey, java.lang.Iterable 
lines,
  Reducer.Context context)
  throws java.io.IOException, InterruptedException
{code}
Inside method, each line is parsed and in case of bad line, the method returns 
from there , instead of continuing with next line. So all subsequent data are 
getting ignored. 

{code}
catch (ImportTsv.TsvParser.BadTsvLineException badLine) {
  if (skipBadLines) {
System.err.println("Bad line." + badLine.getMessage());
incrementBadLineCount(1);
return;
  }
{code}

> Correct data also getting skipped along with bad data in importTsv bulk load 
> thru TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14365) Error scanning 'labels' table in logs with exception while running bulkload, even visibility feature is disabled

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734839#comment-14734839
 ] 

Bhupendra Kumar Jain commented on HBASE-14365:
--

Test case failures are not related to patch.

> Error scanning 'labels' table in logs with exception while running bulkload, 
> even visibility feature is disabled
> 
>
> Key: HBASE-14365
> URL: https://issues.apache.org/jira/browse/HBASE-14365
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14365.patch
>
>
> If visibility feature is disabled, Below exception in logs during importtsv 
> run. In case feature is disabled, No need to log the below as error. Its is 
> bit misleading for the user.
> {code}
> ERROR [main] 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver: Error 
> scanning 'labels' table
> org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:858)
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:756)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:1)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:184)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:311)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:286)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:162)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at 
> org.apache.hadoop.hbase.client.ClientSimpleScanner.(ClientSimpleScanner.java:42)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:381)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: 0001-HBASE-14366_1.patch

Also handled invalid visibility label case as bad line.  Please review

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14380:
-
Status: Patch Available  (was: Open)

> Correct data also getting skipped along with bad data in importTsv bulk load 
> thru TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Attachments: 0001-HBASE-14380.patch
>
>
> Cosider the input data is as below 
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1   >> Correct line
> r1 >> Bad line
> r1,3,v3   >> Correct line
> r1,4,v4   >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper 
> ,  All the lines are getting ignored even though skipBadLines is set to true. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Status: Patch Available  (was: Open)

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14365) Error scanning 'labels' table in logs with exception while running bulkload, even visibility feature is disabled

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14365:
-
Status: Patch Available  (was: Open)

> Error scanning 'labels' table in logs with exception while running bulkload, 
> even visibility feature is disabled
> 
>
> Key: HBASE-14365
> URL: https://issues.apache.org/jira/browse/HBASE-14365
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14365.patch
>
>
> If visibility feature is disabled, Below exception in logs during importtsv 
> run. In case feature is disabled, No need to log the below as error. Its is 
> bit misleading for the user.
> {code}
> ERROR [main] 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver: Error 
> scanning 'labels' table
> org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:858)
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:756)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:1)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:184)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:311)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:286)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:162)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at 
> org.apache.hadoop.hbase.client.ClientSimpleScanner.(ClientSimpleScanner.java:42)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:381)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14365) Error scanning 'labels' table in logs with exception while running bulkload, even visibility feature is disabled

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14365:
-
Attachment: 0001-HBASE-14365.patch

Simple patch to ignore the table not found exception and just log message. 

After the patch message in the log will be as below
{code}WARN [main] 
org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver: 'labels' 
table is not found. Visibility Labels feature is not used.{code}

> Error scanning 'labels' table in logs with exception while running bulkload, 
> even visibility feature is disabled
> 
>
> Key: HBASE-14365
> URL: https://issues.apache.org/jira/browse/HBASE-14365
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14365.patch
>
>
> If visibility feature is disabled, Below exception in logs during importtsv 
> run. In case feature is disabled, No need to log the below as error. Its is 
> bit misleading for the user.
> {code}
> ERROR [main] 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver: Error 
> scanning 'labels' table
> org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:858)
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:756)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:1)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:184)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:311)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:286)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:162)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at 
> org.apache.hadoop.hbase.client.ClientSimpleScanner.(ClientSimpleScanner.java:42)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:381)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-14366:
-
Attachment: 0001-HBASE-14366.patch

Added Null check. Please review the patch. 

After the change, message in log will be "Invalid visibility label "

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain reassigned HBASE-14366:


Assignee: Bhupendra Kumar Jain

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14365) Error scanning 'labels' table in logs with exception while running bulkload, even visibility feature is disabled

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain reassigned HBASE-14365:


Assignee: Bhupendra Kumar Jain

> Error scanning 'labels' table in logs with exception while running bulkload, 
> even visibility feature is disabled
> 
>
> Key: HBASE-14365
> URL: https://issues.apache.org/jira/browse/HBASE-14365
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
>
> If visibility feature is disabled, Below exception in logs during importtsv 
> run. In case feature is disabled, No need to log the below as error. Its is 
> bit misleading for the user.
> ERROR [main] 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver: Error 
> scanning 'labels' table
> org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:858)
> at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:756)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:1)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:184)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:311)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:286)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:162)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at 
> org.apache.hadoop.hbase.client.ClientSimpleScanner.(ClientSimpleScanner.java:42)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:381)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731194#comment-14731194
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

Yes, there is check. In case ZERO ordinal value, further code throws 
InvalidLabelException.  The cell will be dropped in case of invalid vis label. 


> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731221#comment-14731221
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

In case of invalid vis label,  actually the current running map or reducer task 
will terminate. So I noticed one behavior inconsistency between TsvImportMapper 
and TextSortReducer. 

TsvImportMapper : one map task handles one tsv line, so one tsv line will be 
dropped. 
TextSortReducer: one reducer task will handle multiple tsv lines having same 
row key (columns with multiple versions), So if any one vis label is invalid, 
it will terminate the reducer task. So here all the tsv lines of that row key 
will be dropped.  But only the line with invalid vis label should be dropped. 

I will create a different JIRA to handle this particular inconsistency 
scenario. 

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13153) enable bulkload to support replication

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14728572#comment-14728572
 ] 

Bhupendra Kumar Jain commented on HBASE-13153:
--

Having cluster id as part of hfile meta data is really nice to have. This meta 
data can clearly indicate the cluster source. 

But during replication, with this appraoch,the cluster id needs to be added to 
each hfile meta block. This will require rewriting of each hfile meta block, so 
we think this will slow down the replication process compare to writing cluster 
id in zk node.  

Also during replication process, when replication end point detect the cycle, 
it needs to refer to hfile meta data. Consider the case where hfile is in 
archive, So I think there is no meta information available for archive file in 
cache. This may take more time too. Please correct if I got it wrong ?

> enable bulkload to support replication
> --
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13153) enable bulkload to support replication

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727319#comment-14727319
 ] 

Bhupendra Kumar Jain commented on HBASE-13153:
--

Thanks all for the review and nice comments. 
[~apurtell]
bq. Since cyclic replication topologies are supported today I think we'd need 
that handled for the bulk load case too or it will lead to subtle and not so 
subtle problems for users.
To detect the cyclic replication case, we will make use of hbase cluster's 
unique id. [ Same as WAL replication]
The unique cluster id of all source hbase clusters will be persisted in ZK of 
Peer cluster under hfile replication node.

For an example ->   c1->c2->c3->c1 is the cyclic replication case
So when file f1 is bulk loaded to c1 cluster and then from c1->c2 and c2->c3, 
below is the sample zk node data
||Cluster||hfile node data
|c1|f1,{NONE}
|c2|f1,{c1}
|c3|f1,{c1,c2}

When c3 tries to replicate the f1 to c1, it will detect the cycle and will not 
process further. Unique cluster id of all sources will be passed to next 
replication request. 

bq. A crazy idea: rather than have bulk load tooling produce only HFiles for 
replication, why not HFiles for the local cluster and ready made WALs to queue 
up for replication? Of course that's going to have some drawbacks too but I 
think fewer.
We thought similar ideas initially, but didn't take this approach because,
This way we will not get the benefit of bulk load. If we simulate the bulk load 
hfile replication as WAL, then it will actually become many Puts in peer 
cluster and not bulk load. But as per our approach, the hfile will be copied 
and loaded to peer cluster similar as Complete Bulk Load flow, so it will have 
same benefit of bulk load mechanism.


[~tedyu]
bq. Could sequence Id be used so that the HFiles don't need to be written again 
?
As we think, To detect the cycle, Sequence ID can not be used because Sequence 
id's for hfile will be different across clusters and it doesn't provide any 
hint of source cluster.

[~ram_krish]
bq. Few things to consider, ensure that if there is block encoding then the 
encoding scheme is same in both the tables. These type of conditions may come 
in the initial pre checks that we may need to add.
This scenario is similar to changing the encoding in one running hbase cluster. 
Some hfiles will be of X encoding and others will be of Y encoding. Each hfile 
is aware of its encoding type. As I know, this is already handled as part of 
hfile read. So replication of hfile should not have any issue. Correct me if I 
am missing anything ? 

> enable bulkload to support replication
> --
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13978) Variable never assigned in SimpleTotalOrderPartitioner.getPartition()

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-13978:
-
Attachment: 0001-HBASE-13978-Variable-never-assigned-in-SimpleTotalOr.patch

Simple patch. Please review. 

 Variable never assigned in SimpleTotalOrderPartitioner.getPartition() 
 --

 Key: HBASE-13978
 URL: https://issues.apache.org/jira/browse/HBASE-13978
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Bhupendra Kumar Jain
  Labels: beginner
 Fix For: 2.0.0, 1.2.0

 Attachments: 
 0001-HBASE-13978-Variable-never-assigned-in-SimpleTotalOr.patch


 See 
 https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SimpleTotalOrderPartitioner.java#L104,
  which has an {{if}} statement that tries to limit the code to run only once, 
 but since it does not assign {{this.lastReduces}} it will always trigger and 
 recompute the splits (and log them).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13978) Variable never assigned in SimpleTotalOrderPartitioner.getPartition()

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-13978:
-
Fix Version/s: 1.3.0
   Status: Patch Available  (was: Open)

 Variable never assigned in SimpleTotalOrderPartitioner.getPartition() 
 --

 Key: HBASE-13978
 URL: https://issues.apache.org/jira/browse/HBASE-13978
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Bhupendra Kumar Jain
  Labels: beginner
 Fix For: 2.0.0, 1.2.0, 1.3.0

 Attachments: 
 0001-HBASE-13978-Variable-never-assigned-in-SimpleTotalOr.patch


 See 
 https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SimpleTotalOrderPartitioner.java#L104,
  which has an {{if}} statement that tries to limit the code to run only once, 
 but since it does not assign {{this.lastReduces}} it will always trigger and 
 recompute the splits (and log them).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-13982) Add info for visibility labels/cell TTLs to ImportTsv

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain reassigned HBASE-13982:


Assignee: Bhupendra Kumar Jain

 Add info for visibility labels/cell TTLs to ImportTsv
 -

 Key: HBASE-13982
 URL: https://issues.apache.org/jira/browse/HBASE-13982
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Bhupendra Kumar Jain
  Labels: beginner
 Fix For: 2.0.0, 1.2.0


 HBASE-9832 added support for two more optional, special TSV columns, but no 
 usage info was added. Add.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-13978) Variable never assigned in SimpleTotalOrderPartitioner.getPartition()

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain reassigned HBASE-13978:


Assignee: Bhupendra Kumar Jain

 Variable never assigned in SimpleTotalOrderPartitioner.getPartition() 
 --

 Key: HBASE-13978
 URL: https://issues.apache.org/jira/browse/HBASE-13978
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Bhupendra Kumar Jain
  Labels: beginner
 Fix For: 2.0.0, 1.2.0


 See 
 https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SimpleTotalOrderPartitioner.java#L104,
  which has an {{if}} statement that tries to limit the code to run only once, 
 but since it does not assign {{this.lastReduces}} it will always trigger and 
 recompute the splits (and log them).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13823) Procedure V2: unnecessaery operaions on AssignmentManager#recoverTableInDisablingState() and recoverTableInEnablingState()

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597430#comment-14597430
 ] 

Bhupendra Kumar Jain commented on HBASE-13823:
--

In ActiveProcedureIterator class fetchNext() method 
{code}Procedure proc = null;
  for (; index  (threads.length - 1)  proc != null; ++index) {
proc = ((Executor)threads[index]).getActiveProcedure();
  }
{code}
The for loop condition will be always false as proc is null and it will never 
enter into loop. I think it should be as {code}for (; index  (threads.length - 
1)  proc == null; ++index) {code}

 Procedure V2: unnecessaery operaions on 
 AssignmentManager#recoverTableInDisablingState() and 
 recoverTableInEnablingState()
 --

 Key: HBASE-13823
 URL: https://issues.apache.org/jira/browse/HBASE-13823
 Project: HBase
  Issue Type: Sub-task
  Components: master, proc-v2
Affects Versions: 2.0.0, 1.2.0
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Attachments: HBASE-13823-v0.patch


 AssignmentManager#recoverTableInDisablingState() and 
 AssignmentManager#recoverTableInEnablingState try to complete unfinished 
 enable/disable table operations.  In the past, it is necessary, as master 
 failure could leave table in bad state.  With HBASE-13211, enable/disable 
 operations would be auto-recover by Procedure-V2 logic.  Those recovery 
 operation is not necessary: we can either remove those recovery operation or 
 not replay enable/disable operations in procedure queue.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13932) Add mob integrity check in HFilePrettyPrinter

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597353#comment-14597353
 ] 

Bhupendra Kumar Jain commented on HBASE-13932:
--

LGTM .. Thanks for addressing comments ... 

 Add mob integrity check in HFilePrettyPrinter
 -

 Key: HBASE-13932
 URL: https://issues.apache.org/jira/browse/HBASE-13932
 Project: HBase
  Issue Type: Sub-task
  Components: mob
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
 Fix For: hbase-11339

 Attachments: HBASE-13932-V2.patch, HBASE-13932-V3.patch, 
 HBASE-13932-V4.patch, HBASE-13932.patch


 We need to know whether a reference cell is dangling in mob. We can add this 
 to HFilePrettyPrinter.
 We can add a new option to the command, to check the integrity of mob cells 
 either per region or per file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13932) Add mob integrity check in HFilePrettyPrinter

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597353#comment-14597353
 ] 

Bhupendra Kumar Jain commented on HBASE-13932:
--

LGTM .. Thanks for addressing comments ... 

 Add mob integrity check in HFilePrettyPrinter
 -

 Key: HBASE-13932
 URL: https://issues.apache.org/jira/browse/HBASE-13932
 Project: HBase
  Issue Type: Sub-task
  Components: mob
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
 Fix For: hbase-11339

 Attachments: HBASE-13932-V2.patch, HBASE-13932-V3.patch, 
 HBASE-13932-V4.patch, HBASE-13932.patch


 We need to know whether a reference cell is dangling in mob. We can add this 
 to HFilePrettyPrinter.
 We can add a new option to the command, to check the integrity of mob cells 
 either per region or per file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13932) Add mob integrity check in HFilePrettyPrinter

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593317#comment-14593317
 ] 

Bhupendra Kumar Jain commented on HBASE-13932:
--

Thanks for addressing comments. New patch lgtm. 

Just one more change , can be done before commit.

1) evictMobFilesIfNecessary(foundMobFiles, 50);
 foundMobFiles are initalized with 100. So limit should be 100 or initialize 
 with 50

2) Query : As per current patch, cache scope is only per HFile. Can it be 
across HFile ( Global scope ) ?

 Add mob integrity check in HFilePrettyPrinter
 -

 Key: HBASE-13932
 URL: https://issues.apache.org/jira/browse/HBASE-13932
 Project: HBase
  Issue Type: Sub-task
  Components: mob
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
 Fix For: hbase-11339

 Attachments: HBASE-13932-V2.patch, HBASE-13932.patch


 We need to know whether a reference cell is dangling in mob. We can add this 
 to HFilePrettyPrinter.
 We can add a new option to the command, to check the integrity of mob cells 
 either per region or per file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13932) Add mob integrity check in HFilePrettyPrinter

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591663#comment-14591663
 ] 

Bhupendra Kumar Jain commented on HBASE-13932:
--

Few observations.
1) locations = new ArrayListPath();
 Better initialize with size 2 (as of now only 2 locations)

2) fs.exists(mobFilePath)
 exists call may be costly (Not very sure). Should we have cache for 
 mobFilePath.(Can be a HasHSetPath mobFileExistsCache)
   Many of the cells may be pointing to same mobFilePath. We can utilize this 
cache and avoid exists call.

3) String mobFileName = MobUtils.getMobFileName(cell);
 Any chance that mobFileName becomes NULL or Empty String ? I think, the case 
 will not happen in practical until unless the hfile is corrupted. If the 
 case happens, then new Path(location, mobFileName); will throw 
 IllegalArgumentException. Do we need the NULL / Empty String check along 
 with Error printing.

Please excuse if observations are not correct.  

 Add mob integrity check in HFilePrettyPrinter
 -

 Key: HBASE-13932
 URL: https://issues.apache.org/jira/browse/HBASE-13932
 Project: HBase
  Issue Type: Sub-task
  Components: mob
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
 Fix For: hbase-11339

 Attachments: HBASE-13932.patch


 We need to know whether a reference cell is dangling in mob. We can add this 
 to HFilePrettyPrinter.
 We can add a new option to the command, to check the integrity of mob cells 
 either per region or per file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572786#comment-14572786
 ] 

Bhupendra Kumar Jain commented on HBASE-13702:
--

Yes you are right. For bulk mode its going to run all.  

 ImportTsv: Add dry-run functionality and log bad rows
 -

 Key: HBASE-13702
 URL: https://issues.apache.org/jira/browse/HBASE-13702
 Project: HBase
  Issue Type: New Feature
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
 Attachments: HBASE-13702.patch


 ImportTSV job skips bad records by default (keeps a count though). 
 -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is 
 encountered. 
 To be easily able to determine which rows are corrupted in an input, rather 
 than failing on one row at a time seems like a good feature to have.
 Moreover, there should be 'dry-run' functionality in such kinds of tools, 
 which can essentially does a quick run of tool without making any changes but 
 reporting any errors/warnings and success/failure.
 To identify corrupted rows, simply logging them should be enough. In worst 
 case, all rows will be logged and size of logs will be same as input size, 
 which seems fine. However, user might have to do some work figuring out where 
 the logs. Is there some link we can show to the user when the tool starts 
 which can help them with that?
 For the dry run, we can simply use if-else to skip over writing out KVs, and 
 any other mutations, if present.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570867#comment-14570867
 ] 

Bhupendra Kumar Jain commented on HBASE-13702:
--

As per current patch, dry-run executes only Map task, so its useful only when 
Map task is having lot of extra code logic (parsing, validating, transformation 
etc... ). Dry run can execute that logic and output the errors. 

But there might be many logic present in Combiner, Reducer phase also, Which 
dry-run will not check. So I think better to rename the dry-run function as 
*dry-run-map*. It will be much clear. 

 ImportTsv: Add dry-run functionality and log bad rows
 -

 Key: HBASE-13702
 URL: https://issues.apache.org/jira/browse/HBASE-13702
 Project: HBase
  Issue Type: New Feature
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
 Attachments: HBASE-13702.patch


 ImportTSV job skips bad records by default (keeps a count though). 
 -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is 
 encountered. 
 To be easily able to determine which rows are corrupted in an input, rather 
 than failing on one row at a time seems like a good feature to have.
 Moreover, there should be 'dry-run' functionality in such kinds of tools, 
 which can essentially does a quick run of tool without making any changes but 
 reporting any errors/warnings and success/failure.
 To identify corrupted rows, simply logging them should be enough. In worst 
 case, all rows will be logged and size of logs will be same as input size, 
 which seems fine. However, user might have to do some work figuring out where 
 the logs. Is there some link we can show to the user when the tool starts 
 which can help them with that?
 For the dry run, we can simply use if-else to skip over writing out KVs, and 
 any other mutations, if present.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562591#comment-14562591
 ] 

Bhupendra Kumar Jain commented on HBASE-13702:
--

What is the scope of dry-run functionality ? 

As per current patch , in dry-run , same map task is getting executed. which 
internally performs various operations such as ( Parsing text data, creating 
PUT object, creating Cell object , tags etc. ) .. These operations will consume 
some extra time and actually not required by dry-run functionality ..  I think 
Dry-run should finish very fast. 

If dry-run scope is only to validate the parsing of data, then I think better 
to have a new Map task for dry-run 

 ImportTsv: Add dry-run functionality and log bad rows
 -

 Key: HBASE-13702
 URL: https://issues.apache.org/jira/browse/HBASE-13702
 Project: HBase
  Issue Type: New Feature
Reporter: Apekshit Sharma
Assignee: Apekshit Sharma
 Attachments: HBASE-13702.patch


 ImportTSV job skips bad records by default (keeps a count though). 
 -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is 
 encountered. 
 To be easily able to determine which rows are corrupted in an input, rather 
 than failing on one row at a time seems like a good feature to have.
 Moreover, there should be 'dry-run' functionality in such kinds of tools, 
 which can essentially does a quick run of tool without making any changes but 
 reporting any errors/warnings and success/failure.
 To identify corrupted rows, simply logging them should be enough. In worst 
 case, all rows will be logged and size of logs will be same as input size, 
 which seems fine. However, user might have to do some work figuring out where 
 the logs. Is there some link we can show to the user when the tool starts 
 which can help them with that?
 For the dry run, we can simply use if-else to skip over writing out KVs, and 
 any other mutations, if present.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5950) Add a decimal comparator for Filter

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541953#comment-14541953
 ] 

Bhupendra Kumar Jain commented on HBASE-5950:
-

I think, For Numeric comparison of positive and negative value, Binary 
comparator will not be sufficient enough. Consider below scenarios
{Quote}
// Scenario 1 : Compare Negative double value with Negative double value 
using Binary Comparator
double d1 = -4.0d;
double d2 = -5.0d;
BinaryComparator cp = new BinaryComparator(Bytes.toBytes(d1));
assertEquals(1, cp.compareTo(Bytes.toBytes(d2))); //FAIL


// Scenario 2 : Compare Negative long value with Negative long value using 
Binary Comparator
long l1 = -4L;
long l2 = -5L;
BinaryComparator cp1 = new BinaryComparator(Bytes.toBytes(l1));
assertEquals(1, cp1.compareTo(Bytes.toBytes(l2))); //PASS
 

// Scenario 3 : Compare positive long value with Negative long value using 
Binary Comparator
long l3 = 4L;
long l4 = -5L;
BinaryComparator cp2 = new BinaryComparator(Bytes.toBytes(l3));
assertEquals(1, cp2.compareTo(Bytes.toBytes(l4))); //FAIL


// Scenario 4 : Compare positive long value with Negative long value using 
Long Comparator
long l5 = 4L;
long l6 = -5L;
LongComparator cp3 = new LongComparator(l5);
assertEquals(1, cp3.compareTo(Bytes.toBytes(l6))); //PASS
{Quote}

To compare the Non decimal values, LongComparator is sufficient. Similar way to 
compare the Decimal Values , DecimalComparator will be required. 
May be there is some other better way which I am not aware   Am I missing 
something ? 


 Add a decimal comparator for Filter
 ---

 Key: HBASE-5950
 URL: https://issues.apache.org/jira/browse/HBASE-5950
 Project: HBase
  Issue Type: New Feature
  Components: Filters
Affects Versions: 0.94.0, 0.95.2
Reporter: Jieshan Bean
Assignee: Jieshan Bean

 Suppose we have a requirement like below:
 we want to get the rows with one specified column value larger than A and 
 less than B.
 (They are all decimals or integers)
 namely: 
A  Integer.valueof(column)  B
 Use BinaryComparator will not help us to archive that goal:
 e.g.   suppose A = 100, B = 200, one column value is 11.
So it can satisfy that condition, but it's not the row we wanted.
  
 So I suggest to add one comparator to help implementing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13051) Custom line separator option for bulk loading

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345009#comment-14345009
 ] 

Bhupendra Kumar Jain commented on HBASE-13051:
--

As I understand request is for Record Delimiter option.

*ImportTsv* internally uses *TextInputFormat* as input format class which has 
configuration for record delimiter as *textinputformat.record.delimiter* So 
configuring this parameter will be sufficient for your requirement. 
But as a proper solution It will be better, if ImportTsv takes its own 
parameter such as as *importtsv.record.separator*.and internally configures to 
TextInputFormat OR some other input format class in future.  

Whats your opinion ?

 Custom line separator option for bulk loading
 -

 Key: HBASE-13051
 URL: https://issues.apache.org/jira/browse/HBASE-13051
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.10.1
Reporter: sivakumar

 While bulk loading data through ImportTsv dont have an option to choose 
 custom line separator. It defaults to new line character (\n). This request 
 is created to enhance the utility to support custom line separator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-13026) Wrong error message in case incorrect snapshot name OR Incorrect table name

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain reassigned HBASE-13026:


Assignee: Bhupendra Kumar Jain

 Wrong error message in case incorrect snapshot name OR Incorrect table name
 ---

 Key: HBASE-13026
 URL: https://issues.apache.org/jira/browse/HBASE-13026
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bhupendra Kumar Jain
Assignee: Bhupendra Kumar Jain
Priority: Minor
 Fix For: 1.0.0, 2.0.0, 0.98.11

 Attachments: 
 0001-HBASE-13026-Wrong-error-message-in-case-incorrect-sn.patch


 hbase(main):009:0 snapshot 't1', '.t1Snapshot'
 ERROR: Illegal first character 46 at 0. {color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: t1Snapshot
 hbase(main):008:0 create '-test' , 'cf1'
 ERROR: Illegal first character 45 at 0.{color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: -test
  As per message Namespaces is wrong. But in this scenario, snapshot / 
  table name start character is wrong.
 Its because in the code the message is as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[0] +
   at 0. Namespaces can only start 
 with alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName));
 {code}
 The correct code should be as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[start] +
   at 0.  + (isSnapshot ? 
 Snapshot : User-space table) +
   qualifiers can only start with 
 'alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName, start, 
 end));
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13049) wal_roll ruby command doesn't work.

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-13049:
-
Attachment: 0001-HBASE-13049-wal_roll-ruby-command-doesn-t-work.patch

 wal_roll ruby command doesn't work. 
 

 Key: HBASE-13049
 URL: https://issues.apache.org/jira/browse/HBASE-13049
 Project: HBase
  Issue Type: Bug
  Components: shell
Reporter: Bhupendra Kumar Jain
 Attachments: 0001-HBASE-13049-wal_roll-ruby-command-doesn-t-work.patch


 On execution of wal_roll command in shell, error message gets displayed as 
 shown below
 hbase(main):005:0 wal_roll 'host-10-19-92-94,16201,1424081618286'
 *ERROR: cannot convert instance of class org.jruby.RubyString to class 
 org.apache.hadoop.hbase.ServerName*
 its because Admin Java api expecting a ServerName object but script passes 
 the ServerName as string. 
 currently script is as below
 {code}
 @admin.rollWALWriter(server_name)
 {code}
 It should be like 
 {code}
 @admin.rollWALWriter(ServerName.valueOf(server_name))
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13049) wal_roll ruby command doesn't work.

[Bhupendra Kumar Jain (JIRA)]

Bhupendra Kumar Jain created HBASE-13049:


 Summary: wal_roll ruby command doesn't work. 
 Key: HBASE-13049
 URL: https://issues.apache.org/jira/browse/HBASE-13049
 Project: HBase
  Issue Type: Bug
  Components: shell
Reporter: Bhupendra Kumar Jain


On execution of wal_roll command in shell, error message gets displayed as 
shown below

hbase(main):005:0 wal_roll 'host-10-19-92-94,16201,1424081618286'

*ERROR: cannot convert instance of class org.jruby.RubyString to class 
org.apache.hadoop.hbase.ServerName*

its because Admin Java api expecting a ServerName object but script passes the 
ServerName as string. 
currently script is as below
{code}
@admin.rollWALWriter(server_name)
{code}

It should be like 
{code}
@admin.rollWALWriter(ServerName.valueOf(server_name))
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13049) wal_roll ruby command doesn't work.

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-13049:
-
Status: Patch Available  (was: Open)

Patch attached. Please review. 

 wal_roll ruby command doesn't work. 
 

 Key: HBASE-13049
 URL: https://issues.apache.org/jira/browse/HBASE-13049
 Project: HBase
  Issue Type: Bug
  Components: shell
Reporter: Bhupendra Kumar Jain
 Attachments: 0001-HBASE-13049-wal_roll-ruby-command-doesn-t-work.patch


 On execution of wal_roll command in shell, error message gets displayed as 
 shown below
 hbase(main):005:0 wal_roll 'host-10-19-92-94,16201,1424081618286'
 *ERROR: cannot convert instance of class org.jruby.RubyString to class 
 org.apache.hadoop.hbase.ServerName*
 its because Admin Java api expecting a ServerName object but script passes 
 the ServerName as string. 
 currently script is as below
 {code}
 @admin.rollWALWriter(server_name)
 {code}
 It should be like 
 {code}
 @admin.rollWALWriter(ServerName.valueOf(server_name))
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13026) Wrong error message in case incorrect snapshot name OR Incorrect table name

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-13026:
-
Attachment: 0001-HBASE-13026-Wrong-error-message-in-case-incorrect-sn.patch

 Wrong error message in case incorrect snapshot name OR Incorrect table name
 ---

 Key: HBASE-13026
 URL: https://issues.apache.org/jira/browse/HBASE-13026
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bhupendra Kumar Jain
Priority: Minor
 Attachments: 
 0001-HBASE-13026-Wrong-error-message-in-case-incorrect-sn.patch


 hbase(main):009:0 snapshot 't1', '.t1Snapshot'
 ERROR: Illegal first character 46 at 0. {color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: t1Snapshot
 hbase(main):008:0 create '-test' , 'cf1'
 ERROR: Illegal first character 45 at 0.{color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: -test
  As per message Namespaces is wrong. But in this scenario, snapshot / 
  table name start character is wrong.
 Its because in the code the message is as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[0] +
   at 0. Namespaces can only start 
 with alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName));
 {code}
 The correct code should be as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[start] +
   at 0.  + (isSnapshot ? 
 Snapshot : User-space table) +
   qualifiers can only start with 
 'alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName, start, 
 end));
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13026) Wrong error message in case incorrect snapshot name OR Incorrect table name

[Bhupendra Kumar Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupendra Kumar Jain updated HBASE-13026:
-
Status: Patch Available  (was: Open)

 Wrong error message in case incorrect snapshot name OR Incorrect table name
 ---

 Key: HBASE-13026
 URL: https://issues.apache.org/jira/browse/HBASE-13026
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bhupendra Kumar Jain
Priority: Minor
 Attachments: 
 0001-HBASE-13026-Wrong-error-message-in-case-incorrect-sn.patch


 hbase(main):009:0 snapshot 't1', '.t1Snapshot'
 ERROR: Illegal first character 46 at 0. {color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: t1Snapshot
 hbase(main):008:0 create '-test' , 'cf1'
 ERROR: Illegal first character 45 at 0.{color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: -test
  As per message Namespaces is wrong. But in this scenario, snapshot / 
  table name start character is wrong.
 Its because in the code the message is as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[0] +
   at 0. Namespaces can only start 
 with alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName));
 {code}
 The correct code should be as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[start] +
   at 0.  + (isSnapshot ? 
 Snapshot : User-space table) +
   qualifiers can only start with 
 'alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName, start, 
 end));
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13026) Wrong error message in case incorrect snapshot name OR Incorrect table name

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317965#comment-14317965
 ] 

Bhupendra Kumar Jain commented on HBASE-13026:
--

Patch file attached. Please have a look.

 Wrong error message in case incorrect snapshot name OR Incorrect table name
 ---

 Key: HBASE-13026
 URL: https://issues.apache.org/jira/browse/HBASE-13026
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bhupendra Kumar Jain
Priority: Minor
 Attachments: 
 0001-HBASE-13026-Wrong-error-message-in-case-incorrect-sn.patch


 hbase(main):009:0 snapshot 't1', '.t1Snapshot'
 ERROR: Illegal first character 46 at 0. {color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: t1Snapshot
 hbase(main):008:0 create '-test' , 'cf1'
 ERROR: Illegal first character 45 at 0.{color:red}*Namespaces*{color} can 
 only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: -test
  As per message Namespaces is wrong. But in this scenario, snapshot / 
  table name start character is wrong.
 Its because in the code the message is as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[0] +
   at 0. Namespaces can only start 
 with alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName));
 {code}
 The correct code should be as below
 {code}
 if (qualifierName[start] == '.' || qualifierName[start] == '-') {
   throw new IllegalArgumentException(Illegal first character  + 
 qualifierName[start] +
   at 0.  + (isSnapshot ? 
 Snapshot : User-space table) +
   qualifiers can only start with 
 'alphanumeric  +
  characters': i.e. [a-zA-Z_0-9]:  +
  Bytes.toString(qualifierName, start, 
 end));
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13026) Wrong error message in case incorrect snapshot name OR Incorrect table name

[Bhupendra Kumar Jain (JIRA)]

Bhupendra Kumar Jain created HBASE-13026:


 Summary: Wrong error message in case incorrect snapshot name OR 
Incorrect table name
 Key: HBASE-13026
 URL: https://issues.apache.org/jira/browse/HBASE-13026
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bhupendra Kumar Jain
Priority: Minor


hbase(main):009:0 snapshot 't1', '.t1Snapshot'

ERROR: Illegal first character 46 at 0. {color:red}*Namespaces*{color} can 
only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: t1Snapshot

hbase(main):008:0 create '-test' , 'cf1'

ERROR: Illegal first character 45 at 0.{color:red}*Namespaces*{color} can 
only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: -test

 As per message Namespaces is wrong. But in this scenario, snapshot / table 
 name start character is wrong.

Its because in the code the message is as below
{code}
if (qualifierName[start] == '.' || qualifierName[start] == '-') {
  throw new IllegalArgumentException(Illegal first character  + 
qualifierName[0] +
  at 0. Namespaces can only start 
with alphanumeric  +
 characters': i.e. [a-zA-Z_0-9]:  +
 Bytes.toString(qualifierName));
{code}

The correct code should be as below
{code}
if (qualifierName[start] == '.' || qualifierName[start] == '-') {
  throw new IllegalArgumentException(Illegal first character  + 
qualifierName[start] +
  at 0.  + (isSnapshot ? Snapshot 
: User-space table) +
  qualifiers can only start with 
'alphanumeric  +
 characters': i.e. [a-zA-Z_0-9]:  +
 Bytes.toString(qualifierName, start, 
end));
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)