date:20120616


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296117#comment-13296117
 ] 

Andrew Purtell commented on HBASE-6222:
---

It's rather shocking to see congressional level discussion about  NoSQL winners 
and losers. 

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5838) Add an LZ4 compression option to HFile

2012-06-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296137#comment-13296137
 ] 

Hudson commented on HBASE-5838:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #56 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/56/])
HBASE-5838. Add an LZ4 compression option to HFile (Revision 1350844)

 Result = FAILURE
apurtell : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java


 Add an LZ4 compression option to HFile
 --

 Key: HBASE-5838
 URL: https://issues.apache.org/jira/browse/HBASE-5838
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.96.0, 0.94.1

 Attachments: 5838.patch


 HADOOP-7657 adds support for LZ4 compression to Hadoop core. As with Snappy, 
 we should add reflection based support for this alternative to 
 HFile.Compression. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6165) Replication can overrun .META scans on cluster re-start


[ 
https://issues.apache.org/jira/browse/HBASE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296141#comment-13296141
 ] 

Lars Hofhansl commented on HBASE-6165:
--

What's a good approach to avoid this?

 Replication can overrun .META scans on cluster re-start
 ---

 Key: HBASE-6165
 URL: https://issues.apache.org/jira/browse/HBASE-6165
 Project: HBase
  Issue Type: Bug
Reporter: Elliott Clark

 When restarting a large set of regions on a reasonably small cluster the 
 replication from another cluster tied up every xceiver meaning nothing could 
 be onlined.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

2012-06-16 Thread Lars George (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296151#comment-13296151
 ] 

Lars George commented on HBASE-6222:


I was thinking about this as well a while back when Accumulo came up. In that 
context I thought it might be good to have the option to add tags to each KV. 
This can be used for security or other purposes, like replication or 
transactional support. Another use case is filtering as these tags could have 
special features like caching or indexes. 

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5998) Bulk assignment: regionserver optimization by using a temporary cache for table descriptors when receveing an open regions request

2012-06-16 Thread rajeshbabu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

rajeshbabu updated HBASE-5998:
--

Attachment: HBASE-5998_94.patch

Patch for 94. Please review and provide comments/suggestions.


 Bulk assignment: regionserver optimization by using a temporary cache for 
 table descriptors when receveing an open regions request
 --

 Key: HBASE-5998
 URL: https://issues.apache.org/jira/browse/HBASE-5998
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.96.0

 Attachments: 5998.v2.patch, 5998.v3.patch, HBASE-5998_94.patch


 During the assignment, on the regionserver, before creating the handlers we 
 load the table description. Even if there is a cache, we check the timestamps 
 for each region, while it's not necessary. The test below is just with one 
 node, with more nodes the benefit will improve. By limiting the time spent in 
 HRegion#openRegion we increase the parallelization during cluster startup, as 
 the master is using a pool of threads to call the RS.
 -- Without the fix
 2012-05-14 11:40:52,501 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
 region(s) to localhost,11003,1336988444043
 2012-05-14 11:41:09,947 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
 localhost,11003,1336988444043
 -- With the fix
 2012-05-14 11:34:40,444 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
 region(s) to localhost,11003,1336988444043
 2012-05-14 11:34:40,929 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
 localhost,11003,1336988065948

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296177#comment-13296177
 ] 

Andrew Purtell commented on HBASE-6222:
---

HBASE-4477 could transmit tags in the WAL. 

HBASE-2893 Is one way to manage them in the store. 

Revisit those issues?

Others? New approaches?

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296182#comment-13296182
 ] 

Jonathan Hsieh commented on HBASE-6222:
---

Here's a good link with high level semantics of their implementation.  

http://accumulo.apache.org/1.4/user_manual/Security.html

I've spent a few hours looking at the accumulo code.  Internally it is:
* an extra visibility/tags field in their cell Key which supports AND, OR, NOT 
boolean operators.
* an always on visibility filter on the tablet/regionserver side.
* a constraint policy option that determines behavior if a users attempts to 
write to labels they do not have read access to:
** allow
** throw security exceptions
* authentication credentials and authorization plumbing




 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296184#comment-13296184
 ] 

Andrew Purtell commented on HBASE-6222:
---

@Jon we could do it that way but then the changes are core  . A big concern 
when This first came up is the impact increasing the size of every KV would 
have for something that won't be the common case. 

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

2012-06-16 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13296186#comment-13296186
 ] 

Jimmy Xiang commented on HBASE-6222:


We can put some attribute on the table level to control if the feature is 
enabled for a table.  So that in normal case, the KV is not affected.
We can also put such information in a system table.

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13305339#comment-13305339
 ] 

Andrew Purtell commented on HBASE-6222:
---

@Jimmy Such a proposal must address the impact on code. So is that a subclass 
of KV? Or a bunch of conditional code wherever KVs are handled?

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393356#comment-13393356
]

Jonathan Hsieh commented on HBASE-6222:
---

@Andy I'm not proposing we necessarily do it the way accumulo does it -- the
more important point is that we mimic it's semantics to make it easier for
users to eventually port to HBase. :)

The meta column idea you describe in HBASE-2893 and its use cases seems to also
argue for restoring the distinction between locality groups and column
families from original bigtable paper. (these are conflated in HBase).

Using a meta column for cell-granularity visibility seems like a reasonable
hbase implementation option. I'm not convinced why this approach would save
significantly more space. I do think the not changing the not changing core
argument is more compelling constriant.

The accumulo implementation has a default case that would make it cost the tags
cost extra only on cells with non-default behavior (e.g. only the
highly-sensitive or cols with sensitivity varying cells). They delta encode
their keys (including the visiblity tags) like we do in 0.94.

As a straw man, a core-changing HBase version that modified KV's wouldn't
necessarily incur more space -- we'd could a new KeyValue.Type (TaggedPut?)
that had visibility settings, and use the old Put type that used the default
tags.

Add per-KeyValue Security, get federal funding for HBase?
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

Saw an interesting article:
http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
The Senate Armed Services Committee version of the fiscal 2013 national
defense authorization act (S. 3254) would require DoD agencies to foreswear
the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO
certifies that there exists either no viable commercial open source database
with security features comparable to [Accumulo] (such as the HBase or
Cassandra databases)...
Not sure what a 'commercial open source database' is, and I'm not sure whats
going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security'
like Accumulo's, we might put ourselves in the running for federal
contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393356#comment-13393356
]

Jonathan Hsieh edited comment on HBASE-6222 at 6/16/12 5:59 PM:

@Andy I'm not proposing we necessarily do it the way accumulo does it -- the
more important point is that we mimic it's semantics to make it easier for
users to eventually port to HBase. :)

Using a meta column for cell-granularity visibility seems like a reasonable
hbase implementation option. I'm not convinced why this approach would save
significantly more space. I do think the not changing core argument is more
compelling hbase-specific constraint.

was (Author: jmhsieh):
@Andy I'm not proposing we necessarily do it the way accumulo does it --
the more important point is that we mimic it's semantics to make it easier for
users to eventually port to HBase. :)

Add per-KeyValue Security, get federal funding for HBase?
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

[jira] [Comment Edited] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393356#comment-13393356
]

Jonathan Hsieh edited comment on HBASE-6222 at 6/16/12 6:00 PM:

@Andy I'm not proposing we necessarily do it the way accumulo does it -- the
more important point is that we mimic it's semantics to make it easier for
users to eventually port to HBase. :)

Using a meta column for cell-granularity visibility seems like a reasonable
hbase implementation option. I'm not convinced why this approach would save
significantly more space. I do think the not changing core argument is more
compelling hbase-specific constraint.

As a straw man, a core-changing HBase version that modified KV's wouldn't
necessarily incur more space -- we could use a new KeyValue.Type (TaggedPut?)
that had visibility settings, and use the old Put type that used the default
tags.

Using a meta column for cell-granularity visibility seems like a reasonable
hbase implementation option. I'm not convinced why this approach would save
significantly more space. I do think the not changing core argument is more
compelling hbase-specific constraint.

Add per-KeyValue Security, get federal funding for HBase?
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

2012-06-16 Thread Lars George (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393360#comment-13393360
 ] 

Lars George commented on HBASE-6222:


I am thinking of moving the tags between the key and the value of the KV. The 
existence would need to be tracked somehow, so we could add a single byte as a 
bit field to flag the presence. Or add a new type value like PUT_WITH_TAGS or 
some such. That would leave existing KVs the same and only the code needs to be 
enhanced to enable the actual security layer - which would be a system level 
coprocessor I presume. 

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

2012-06-16 Thread Lars George (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393361#comment-13393361
 ] 

Lars George commented on HBASE-6222:


Oh, and that approach would also leave all the comparators etc. untouched. That 
sounds like a good thing. 

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393374#comment-13393374
]

Andrew Purtell commented on HBASE-6222:
---

Thus far we have various hooks and a proposal for an out of band approach:
HBASE-2893, HBASE-4477, API object attributes (HBASE-3811, HBASE-3931, etc.).
The impetus (as I understand it) for an out of band approach is the special
nature of KV and the desire to avoid adding additional conditionals/complexity
in code wherever it touches KV. That is balanced by the performance impact of
an out of band approach, as Jon mentions, without locality groups (a
significant change) there is extra IO. I see this as a more conservative
approach that won't perturb performance sensitive members of our community,
because there is zero overhead unless configuration is toggled on, coprocessors
are loaded, and such.

However I am not arguing for or against any particular approach. The
suggestions by Jon, Lars, and Jimmy are the start of a design proposal but
should be pulled together into an actual proposal (as another JIRA?). This is
my point. Adding even only one byte to every KV has a very significant effect
on storage and IO when the data set is large. Adding a new type avoids that but
what is the impact in the code? And it wouldn't be one new type right, there
would be a tagged analogue for every mutator? A convincing exploration of these
issues would go a long way here IMHO.

Add per-KeyValue Security, get federal funding for HBase?
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

[jira] [Comment Edited] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393374#comment-13393374
]

Andrew Purtell edited comment on HBASE-6222 at 6/16/12 7:33 PM:

Thus far we have various hooks and a proposal for an out of band approach:
HBASE-2893, HBASE-4477, API object attributes (HBASE-3811, HBASE-3931, etc.).
The impetus (as I understand it) for an out of band approach is the special
nature of KV and the desire to avoid adding additional conditionals/complexity
in code wherever it touches KV. That is balanced by the problems an out of band
approach causes for minimizing overheads when tags are present, as Jon
mentions, without locality groups (a significant change) there is extra IO. I
see this as a more conservative approach that won't perturb performance
sensitive members of our community, because there is zero overhead unless
configuration is toggled on, coprocessors are loaded, and such.

was (Author: apurtell):
Thus far we have various hooks and a proposal for an out of band approach:
HBASE-2893, HBASE-4477, API object attributes (HBASE-3811, HBASE-3931, etc.).
The impetus (as I understand it) for an out of band approach is the special
nature of KV and the desire to avoid adding additional conditionals/complexity
in code wherever it touches KV. That is balanced by the performance impact of
an out of band approach, as Jon mentions, without locality groups (a
significant change) there is extra IO. I see this as a more conservative
approach that won't perturb performance sensitive members of our community,
because there is zero overhead unless configuration is toggled on, coprocessors
are loaded, and such.

Add per-KeyValue Security, get federal funding for HBase?
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

[jira] [Comment Edited] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393374#comment-13393374
]

Andrew Purtell edited comment on HBASE-6222 at 6/16/12 7:38 PM:

Thus far we have various hooks and a proposal for an out of band approach:
HBASE-2893, HBASE-4477, API object attributes (HBASE-3811, HBASE-3921,
HBASE-3931, etc.). The impetus (as I understand it) for an out of band approach
is the special nature of KV and the desire to avoid adding additional
conditionals/complexity in code wherever it touches KV. That is balanced by the
problems an out of band approach causes for minimizing overheads when tags are
present, as Jon mentions, without locality groups (a significant change) there
is extra IO. I see this as a more conservative approach that won't perturb
performance sensitive members of our community, because there is zero overhead
unless configuration is toggled on, coprocessors are loaded, and such.

was (Author: apurtell):
Thus far we have various hooks and a proposal for an out of band approach:
HBASE-2893, HBASE-4477, API object attributes (HBASE-3811, HBASE-3931, etc.).
The impetus (as I understand it) for an out of band approach is the special
nature of KV and the desire to avoid adding additional conditionals/complexity
in code wherever it touches KV. That is balanced by the problems an out of band
approach causes for minimizing overheads when tags are present, as Jon
mentions, without locality groups (a significant change) there is extra IO. I
see this as a more conservative approach that won't perturb performance
sensitive members of our community, because there is zero overhead unless
configuration is toggled on, coprocessors are loaded, and such.

Add per-KeyValue Security, get federal funding for HBase?
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

[jira] [Updated] (HBASE-3170) RegionServer confused about empty row keys

2012-06-16 Thread Benoit Sigoure (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoit Sigoure updated HBASE-3170:
--

Affects Version/s: 0.90.0
   0.90.1
   0.90.2
   0.90.3
   0.90.4
   0.90.5
   0.90.6
   0.92.0
   0.92.1

 RegionServer confused about empty row keys
 --

 Key: HBASE-3170
 URL: https://issues.apache.org/jira/browse/HBASE-3170
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, 
 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
Reporter: Benoit Sigoure

 I'm no longer sure about the expected behavior when using an empty row key 
 (e.g. a 0-byte long byte array).  I assumed that this was a legitimate row 
 key, just like having an empty column qualifier is allowed.  But it seems 
 that the RegionServer considers the empty row key to be whatever the first 
 row key is.
 {code}
 Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 
 15:30:09 PDT 2010
 hbase(main):001:0 scan 'tsdb-uid', {LIMIT = 1}
 ROW   COLUMN+CELL 
  
  \x00 column=id:metrics, timestamp=1288375187699, 
 value=foo  
  \x00 column=id:tagk, timestamp=1287522021046, 
 value=bar 
  \x00 column=id:tagv, timestamp=1288111387685, 
 value=qux  
 1 row(s) in 0.4610 seconds
 hbase(main):002:0 get 'tsdb-uid', ''
 COLUMNCELL
  
  id:metrics   timestamp=1288375187699, value=foo  

  id:tagk  timestamp=1287522021046, value=bar  

  id:tagv  timestamp=1288111387685, value=qux  
 
 3 row(s) in 0.0910 seconds
 hbase(main):003:0 get 'tsdb-uid', \000
 COLUMNCELL
  
  id:metrics   timestamp=1288375187699, value=foo  

  id:tagk  timestamp=1287522021046, value=bar  

  id:tagv  timestamp=1288111387685, value=qux  
 
 3 row(s) in 0.0550 seconds
 {code}
 This isn't a parsing problem with the command-line of the shell.  I can 
 reproduce this behavior both with plain Java code and with my asynchbase 
 client.
 Since I don't actually have a row with an empty row key, I expected that the 
 first {{get}} would return nothing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6184) HRegionInfo was null or empty in Meta


 [ 
https://issues.apache.org/jira/browse/HBASE-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-6184:
-

Fix Version/s: (was: 0.94.0)
   0.94.1

 HRegionInfo was null or empty in Meta 
 --

 Key: HBASE-6184
 URL: https://issues.apache.org/jira/browse/HBASE-6184
 Project: HBase
  Issue Type: Bug
  Components: client, io
Affects Versions: 0.94.0
Reporter: jiafeng.zhang
 Fix For: 0.94.1

 Attachments: HBASE-6184.patch


 insert data
 hadoop-0.23.2 + hbase-0.94.0
 2012-06-07 13:09:38,573 WARN  
 [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] 
 Encountered problems when prefetch META table: 
 java.io.IOException: HRegionInfo was null or empty in Meta for hbase_one_col, 
 row=hbase_one_col,09115303780247449149,99
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1482)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1367)
 at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:945)
 at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:801)
 at org.apache.hadoop.hbase.client.HTable.put(HTable.java:776)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:397)
 at com.dinglicom.hbase.HbaseImport.insertData(HbaseImport.java:177)
 at com.dinglicom.hbase.HbaseImport.run(HbaseImport.java:210)
 at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5790) ZKUtil deleteRecursively should be a recoverable operation


 [ 
https://issues.apache.org/jira/browse/HBASE-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5790:
-

Fix Version/s: (was: 0.94.1)
 Release Note: Unscheduling from 0.94.x, because ZK 3.4.x requirement.

 ZKUtil deleteRecursively should be a recoverable operation
 --

 Key: HBASE-5790
 URL: https://issues.apache.org/jira/browse/HBASE-5790
 Project: HBase
  Issue Type: Improvement
Reporter: Jesse Yates
Assignee: Jesse Yates
  Labels: zookeeper
 Fix For: 0.96.0

 Attachments: java_HBASE-5790-v1.patch, java_HBASE-5790.patch


 As of 3.4.3 Zookeeper now has full, multi-operation transaction. This means 
 we can wholesale delete chunks of the zk tree and ensure that we don't have 
 any pesky recursive delete issues where we delete the children of a node, but 
 then a child joins before deletion of the parent. Even without transactions, 
 this should be the behavior, but it is possible to make it much cleaner now 
 that we have this new feature in zk.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

2012-06-16 Thread Matt Corgan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393380#comment-13393380
 ] 

Matt Corgan commented on HBASE-6222:


DataBlockEncoding gives us the opportunity to add features to a KeyValue 
interface but create different implementations that have the option of ignoring 
some of the features.  For HBASE-4676 (PrefixTrie), I need to create an 
interface so that I can store the different parts of a KeyValue in different 
backing arrays: 
https://github.com/hotpads/hbase-prefix-trie/blob/master/src/org/apache/hadoop/hbase/cell/HCell.java

This discussion reminds me of the memstoreTS in the current KeyValue which is 
more of an afterthought and has negative performance implications.  With 
DataBlockEncoding I'm able to treat it as a first class citizen when it is 
needed and encode it away when it's not needed.  I suppose there is a small cpu 
cost, but really small relative to what else is happening.

In the future, I think we're going to want to treat the legacy KeyValue and 
legacy LinkedListKeyValue style data blocks as just another DataBlockEncoding 
(DataBlockEncoding.NONE), and instead do most operations through a KeyValue 
interface.  After a lot of benchmarking and research on compiler optimizations, 
I'm pretty sure we are far away from any performance issues with an interface 
if people are worried about that... i can comment more on that later if anyone 
wants.

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5815) ZKUtil.createWithParents should be transactional


 [ 
https://issues.apache.org/jira/browse/HBASE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5815:
-

Fix Version/s: (was: 0.94.1)

Looks like this requires ZK 3.4. Removing from 3.4. Pull back if you feel 
differently.

 ZKUtil.createWithParents should be transactional
 

 Key: HBASE-5815
 URL: https://issues.apache.org/jira/browse/HBASE-5815
 Project: HBase
  Issue Type: Improvement
Reporter: Jesse Yates
Assignee: Jesse Yates
  Labels: zookeeper
 Fix For: 0.96.0

 Attachments: java_HBASE-5815-v0.patch


 Should solve a lot of tricky corner cases when you create the parent, get an 
 update to watchers (who read ZK, but find no children) and then create the 
 children. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5133) Support zookeeper cluster key related parameters for LoadTestTool


 [ 
https://issues.apache.org/jira/browse/HBASE-5133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5133:
-

Fix Version/s: (was: 0.94.1)

No movement, removing from 0.94.x.

 Support zookeeper cluster key related parameters for LoadTestTool
 -

 Key: HBASE-5133
 URL: https://issues.apache.org/jira/browse/HBASE-5133
 Project: HBase
  Issue Type: Sub-task
  Components: client, regionserver
Reporter: Zhihong Ted Yu
 Fix For: 0.96.0


 HBASE-5124 adds zookeeper cluster key related parameters for LoadTestTool
 This task ports them to TRUNK.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5162) Basic client pushback mechanism

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars Hofhansl updated HBASE-5162:
-

Fix Version/s: (was: 0.94.1)
0.96.0

Is this ready? Moving to 0.96. Feel free to pull back.

Basic client pushback mechanism
---

Key: HBASE-5162
URL: https://issues.apache.org/jira/browse/HBASE-5162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0

Attachments: java_HBASE-5162.patch

The current blocking we do when we are close to some limits (memstores over
the multiplier factor, too many store files, global memstore memory) is bad,
too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
that we need something better.
I did a little brainstorm with Stack, we came up quickly with two solutions:
- Send some exception to the client, like OverloadedException, that's thrown
when some situation happens like getting past the low memory barrier. It
would be thrown when the client gets a handler and does some check while
putting or deleting. The client would treat this a retryable exception but
ideally wouldn't check .META. for a new location. It could be fancy and have
multiple levels of pushback, like send the exception to 25% of the clients,
and then go up if the situation persists. Should be easy to implement but
we'll be using a lot more IO to send the payload over and over again (but at
least it wouldn't sit in the RS's memory).
- Send a message alongside a successful put or delete to tell the client to
slow down a little, this way we don't have to do back and forth with the
payload between the client and the server. It's a cleaner (I think) but more
involved solution.
In every case the RS should do very obvious things to notify the operators of
this situation, through logs, web UI, metrics, etc.
Other ideas?

[jira] [Updated] (HBASE-5509) MR based copier for copying HFiles (trunk version)


 [ 
https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5509:
-

Fix Version/s: (was: 0.94.1)
   (was: 0.96.0)

I am not happy with this. I think other features (like hardlinks) are missing 
to make this scheme viable.
Unscheduling.

 MR based copier for copying HFiles (trunk version)
 --

 Key: HBASE-5509
 URL: https://issues.apache.org/jira/browse/HBASE-5509
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, regionserver
Reporter: Karthik Ranganathan
Assignee: Lars Hofhansl
 Attachments: 5509-v2.txt, 5509.txt


 This copier is a modification of the distcp tool in HDFS. It does the 
 following:
 1. List out all the regions in the HBase cluster for the required table
 2. Write the above out to a file
 3. Each mapper 
3.1 lists all the HFiles for a given region by querying the regionserver
3.2 copies all the HFiles
3.3 outputs success if the copy succeeded, failure otherwise. Failed 
 regions are retried in another loop
 4. Mappers are placed on nodes which have maximum locality for a given region 
 to speed up copying

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2012-06-16 Thread Jesse Yates (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393385#comment-13393385
]

Jesse Yates commented on HBASE-5162:

@Lars not yet - needs some more testing and validation on a real cluster. I've
been a bit busy and haven't got a chance to swing back around.

Basic client pushback mechanism
---

Key: HBASE-5162
URL: https://issues.apache.org/jira/browse/HBASE-5162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0

Attachments: java_HBASE-5162.patch

[jira] [Commented] (HBASE-6214) Backport HBASE-5998 to 94.1


[ 
https://issues.apache.org/jira/browse/HBASE-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393386#comment-13393386
 ] 

Lars Hofhansl commented on HBASE-6214:
--

Who's working on this? :)

 Backport HBASE-5998 to 94.1
 ---

 Key: HBASE-6214
 URL: https://issues.apache.org/jira/browse/HBASE-6214
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Anoop Sam John
 Fix For: 0.94.1




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5923) Cleanup checkAndXXX logic


 [ 
https://issues.apache.org/jira/browse/HBASE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5923:
-

Fix Version/s: (was: 0.94.1)

There's too much discussion around this. Removing from 0.94. Might close as 
won't fix altogether.


 Cleanup checkAndXXX logic
 -

 Key: HBASE-5923
 URL: https://issues.apache.org/jira/browse/HBASE-5923
 Project: HBase
  Issue Type: Improvement
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0

 Attachments: 5923-0.94.txt, 5923-trunk.txt


 1. the checkAnd{Put|Delete} method that takes a CompareOP is not exposed via 
 HTable[Interface].
 2. there is unnecessary duplicate code in the check{Put|Delete} code in 
 HRegionServer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393390#comment-13393390
]

Jonathan Hsieh commented on HBASE-6222:
---

Do we think of the goal of faithfully mimicking accumulo's is reasonable? (my
first impression is yes). If so, I think we'd need to define those semantics
and to do that we'd need bring up an accumulo cluster (or spend more time in
the code) to see what its semantics are.

Here's the accumulo api. (ColumnVisibility is tag expression)
http://accumulo.apache.org/1.4/apidocs/org/apache/accumulo/core/data/Mutation.html

You can see that delete mutations in accumulo also have visibility expressions
-- and I'm not actually sure what the semantics are. Let's say you have a
public put on row A, and then a sensitive delete of the entire row A. Does
someone with the only the public authorizations see the old row A, or no row
A? If it was no row A, would they be able to do a versioned query and get the
old row A (and could then infer out that there was a sensitive delete?)...

Also, I didn't see atomic increment operator in accumulo. If it doesn't exist
we'd have to define that behavior. Some options:
1) No visibility settings on hbase increment mutation operations, thus only
keep only one value always readable by all
2) Add some mechanism to set counter visibility so we could to require
authorization to read.
3) Keep one counter for each visibility tag on a particular cell... (yuck).

Add per-KeyValue Security, get federal funding for HBase?
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

[jira] [Commented] (HBASE-6188) Remove the concept of table owner

2012-06-16 Thread Laxman (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393391#comment-13393391
 ] 

Laxman commented on HBASE-6188:
---

Thanks for pointing it out Andy. I couldn't notice these test failures as they 
are intermittent failures.
Even in QA bot build also passing. I will correct this.

{quote}
The new code in postCreateTable must make a special case for the ACL table. 
It's not possible to call AccessControlLists.addUserPermission before the ACL 
table is deployed, i.e. created.
{quote}

Introducing a check like below is fine?

{code}
  public void postCreateTable(ObserverContextMasterCoprocessorEnvironment c,
  HTableDescriptor desc, HRegionInfo[] regions) throws IOException {
if (!AccessControlLists.isAclTable(desc)) {
  String owner = desc.getOwnerString();
  // default the table owner to current user, if not specified.
  if (owner == null) owner = getActiveUser().getShortName();
  UserPermission userperm = new UserPermission(Bytes.toBytes(owner), 
desc.getName(), null,
  Action.values());
  
AccessControlLists.addUserPermission(c.getEnvironment().getConfiguration(), 
userperm);
}
  }
{code}

Apologies for the noise due to multiple submissions for this issue.

 Remove the concept of table owner
 -

 Key: HBASE-6188
 URL: https://issues.apache.org/jira/browse/HBASE-6188
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.94.0, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Laxman
  Labels: security
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, HBASE-6188.patch


 The table owner concept was a design simplification in the initial drop.
 First, the design changes under review means only a user with GLOBAL CREATE 
 permission can create a table, which will probably be an administrator.
 Then, granting implicit permissions may lead to oversights and it adds 
 unnecessary conditionals to our code. So instead the administrator with 
 GLOBAL CREATE permission should make the appropriate grants at table create 
 time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security, get federal funding for HBase?


[ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393395#comment-13393395
 ] 

Andrew Purtell commented on HBASE-6222:
---

@Matt So attributes carry in information and we use a new encoding type to 
store tags adjacent to KVs if they are labelled?

@Jon For counters definitely not 1, option 2 seems better. 

 Add per-KeyValue Security, get federal funding for HBase?
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6222) Add per-KeyValue Security


 [ 
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6222:
--

Summary: Add per-KeyValue Security  (was: Add per-KeyValue Security, get 
federal funding for HBase?)

 Add per-KeyValue Security
 -

 Key: HBASE-6222
 URL: https://issues.apache.org/jira/browse/HBASE-6222
 Project: HBase
  Issue Type: New Feature
  Components: security
Reporter: stack

 Saw an interesting article: 
 http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
 The  Senate Armed Services Committee version of the fiscal 2013 national 
 defense authorization act (S. 3254) would require DoD agencies to foreswear 
 the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO 
 certifies that there exists either no viable commercial open source database 
 with security features comparable to [Accumulo] (such as the HBase or 
 Cassandra databases)...
 Not sure what a 'commercial open source database' is, and I'm not sure whats 
 going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' 
 like Accumulo's, we might put ourselves in the running for federal 
 contributions?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6188) Remove the concept of table owner


[ 
https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393396#comment-13393396
 ] 

Andrew Purtell commented on HBASE-6188:
---

@Laxman seems reasonable. 

 Remove the concept of table owner
 -

 Key: HBASE-6188
 URL: https://issues.apache.org/jira/browse/HBASE-6188
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.94.0, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Laxman
  Labels: security
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, HBASE-6188.patch


 The table owner concept was a design simplification in the initial drop.
 First, the design changes under review means only a user with GLOBAL CREATE 
 permission can create a table, which will probably be an administrator.
 Then, granting implicit permissions may lead to oversights and it adds 
 unnecessary conditionals to our code. So instead the administrator with 
 GLOBAL CREATE permission should make the appropriate grants at table create 
 time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6188) Remove the concept of table owner

2012-06-16 Thread Laxman (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laxman updated HBASE-6188:
--

Attachment: HBASE-6188.3.patch

Attached the new patch

 Remove the concept of table owner
 -

 Key: HBASE-6188
 URL: https://issues.apache.org/jira/browse/HBASE-6188
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.94.0, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Laxman
  Labels: security
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, 
 HBASE-6188.3.patch, HBASE-6188.patch


 The table owner concept was a design simplification in the initial drop.
 First, the design changes under review means only a user with GLOBAL CREATE 
 permission can create a table, which will probably be an administrator.
 Then, granting implicit permissions may lead to oversights and it adds 
 unnecessary conditionals to our code. So instead the administrator with 
 GLOBAL CREATE permission should make the appropriate grants at table create 
 time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6188) Remove the concept of table owner

2012-06-16 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393412#comment-13393412
]

Hadoop QA commented on HBASE-6188:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12532310/HBASE-6188.3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.security.access.TestAccessController

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2175//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2175//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2175//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2175//console

This message is automatically generated.

Remove the concept of table owner
-

Key: HBASE-6188
URL: https://issues.apache.org/jira/browse/HBASE-6188
Project: HBase
Issue Type: Sub-task
Components: security
Affects Versions: 0.94.0, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Laxman
Labels: security
Fix For: 0.96.0, 0.94.1

Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch,
HBASE-6188.3.patch, HBASE-6188.patch

The table owner concept was a design simplification in the initial drop.
First, the design changes under review means only a user with GLOBAL CREATE
permission can create a table, which will probably be an administrator.
Then, granting implicit permissions may lead to oversights and it adds
unnecessary conditionals to our code. So instead the administrator with
GLOBAL CREATE permission should make the appropriate grants at table create
time.

[jira] [Assigned] (HBASE-6214) Backport HBASE-5998 to 94.1

2012-06-16 Thread rajeshbabu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

rajeshbabu reassigned HBASE-6214:
-

Assignee: rajeshbabu

 Backport HBASE-5998 to 94.1
 ---

 Key: HBASE-6214
 URL: https://issues.apache.org/jira/browse/HBASE-6214
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Anoop Sam John
Assignee: rajeshbabu
 Fix For: 0.94.1




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6214) Backport HBASE-5998 to 94.1

2012-06-16 Thread rajeshbabu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393415#comment-13393415
 ] 

rajeshbabu commented on HBASE-6214:
---

@Lars
Uploaded backport patch for 94.1 in HBASE-5998. Can you look at it?

 Backport HBASE-5998 to 94.1
 ---

 Key: HBASE-6214
 URL: https://issues.apache.org/jira/browse/HBASE-6214
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Anoop Sam John
Assignee: rajeshbabu
 Fix For: 0.94.1




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6222) Add per-KeyValue Security

[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393450#comment-13393450
]

stack commented on HBASE-6222:
--

@Lars When you say ...I thought it might be good to have the option to add
tags to each KV, what would a tag look like? A bit set? Or some
name/values within the KV?

I like Jon's suggestion that HBASE-2893 could be less onerous if we implemented
Locality groups (the meta-cf would be co-mingled w/ the data its meta for).

bq. They delta encode their keys (including the visiblity tags) like we do in
0.94.

Should we turn the above on as default?

bq. As a straw man, a core-changing HBase version that modified KV's wouldn't
necessarily incur more space – we could use a new KeyValue.Type (TaggedPut?)
that had visibility settings, and use the old Put type that used the default
tags.

KVs are also versioned so we should be able to up the version and add new
in-key facility (A while back I messed w/ doing this adding a sequenceid beside
the timestamp... I did it as conditional code rather than as subclass which
for sure compounded the complexity of the already-complex KeyValue --
unfortunately, hopefully there is a better route -- but it seemed possible
making different KV types float in same scanner merge). If we add a
ColumnVisibility-like expression into the KV, we'd have to update Comparators
to exclude this portion from inclusion in sort.

On making KV an Interface, that'd be cool. Todd had a go at it a while back
but the ripple turned into an avalanche of code changes IIRC so he suggested we
do it piecemeal. It'd be sweet though.

+1 on Andrew's proposal first. We need a driver?

On faithful mimicking of Accumulo's implementation, that makes sense in the
absence of an actual customer who needs this stuff (Jon? You have one?)? Maybe
the Accumulo implementation is far from what is wanted -- these 'expression's
are passed in the clear -- and a superior implementation might not be that much
of a stretch?

Add per-KeyValue Security
-

Key: HBASE-6222
URL: https://issues.apache.org/jira/browse/HBASE-6222
Project: HBase
Issue Type: New Feature
Components: security
Reporter: stack

[jira] [Commented] (HBASE-5923) Cleanup checkAndXXX logic


[ 
https://issues.apache.org/jira/browse/HBASE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393451#comment-13393451
 ] 

stack commented on HBASE-5923:
--

bq ...but that would change the wire protocol

Thats fine in 0.96, right?

 Cleanup checkAndXXX logic
 -

 Key: HBASE-5923
 URL: https://issues.apache.org/jira/browse/HBASE-5923
 Project: HBase
  Issue Type: Improvement
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0

 Attachments: 5923-0.94.txt, 5923-trunk.txt


 1. the checkAnd{Put|Delete} method that takes a CompareOP is not exposed via 
 HTable[Interface].
 2. there is unnecessary duplicate code in the check{Put|Delete} code in 
 HRegionServer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5923) Cleanup checkAndXXX logic


[ 
https://issues.apache.org/jira/browse/HBASE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393452#comment-13393452
 ] 

stack commented on HBASE-5923:
--

And, please don't close it as won't fix... This is really nice clean up.

 Cleanup checkAndXXX logic
 -

 Key: HBASE-5923
 URL: https://issues.apache.org/jira/browse/HBASE-5923
 Project: HBase
  Issue Type: Improvement
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.96.0

 Attachments: 5923-0.94.txt, 5923-trunk.txt


 1. the checkAnd{Put|Delete} method that takes a CompareOP is not exposed via 
 HTable[Interface].
 2. there is unnecessary duplicate code in the check{Put|Delete} code in 
 HRegionServer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-6214) Backport HBASE-5998 to 94.1


 [ 
https://issues.apache.org/jira/browse/HBASE-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-6214.
--

  Resolution: Fixed
Hadoop Flags: Reviewed

I committed the patch that was over on hbase-5998.  Looks good to me.  Thanks 
for the backport Rajesh

 Backport HBASE-5998 to 94.1
 ---

 Key: HBASE-6214
 URL: https://issues.apache.org/jira/browse/HBASE-6214
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Anoop Sam John
Assignee: rajeshbabu
 Fix For: 0.94.1




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5998) Bulk assignment: regionserver optimization by using a temporary cache for table descriptors when receveing an open regions request