[jira] [Commented] (HBASE-4478) Improve AssignmentManager.handleRegion so that it can process certain ZK state in the case of RS offline
[
https://issues.apache.org/jira/browse/HBASE-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149004#comment-13149004
]
Hudson commented on HBASE-4478:
---
Integrated in HBase-TRUNK #2431 (See
[https://builds.apache.org/job/HBase-TRUNK/2431/])
HBASE-4478 Improve AssignmentManager.handleRegion so that it can process
certain ZK state
in the case of RS offline
tedyu :
Files :
* /hbase/trunk/CHANGES.txt
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
Improve AssignmentManager.handleRegion so that it can process certain ZK
state in the case of RS offline
Key: HBASE-4478
URL: https://issues.apache.org/jira/browse/HBASE-4478
Project: HBase
Issue Type: Bug
Reporter: Ming Ma
Assignee: ramkrishna.s.vasudevan
Attachments: 4478-v2.txt, HBASE-4478_1.patch
Currently AssignmentManager.handleRegion skips processing of ZK event change
if the RS is offline. It relies on TimeoutMonitor and ServerShutdownHandler
to process RIT.
// Verify this is a known server
if (!serverManager.isServerOnline(sn)
!this.master.getServerName().equals(sn)) {
LOG.warn(Attempted to handle region transition for server but +
server is not online: + Bytes.toString(data.getRegionName()));
return;
}
For certain states like OPENED, OPENING, FAILED_OPEN, CLOSED, it can continue
the progressing even if the RS is offline.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion
[ https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-4768: --- Attachment: D363.3.patch mbautin updated the revision [jira] [HBASE-4768] Per-(table, columnFamily) metrics with configurable table name inclusion. Reviewers: jgray, nspiegelberg, stack, tedyu, todd, JIRA During cluster testing I found a mysterious column family name .tmp. It turns out that with cache-on-write the .tmp directory name was erroneously picked up as column family. Fixing this by setting table name and column family name from Store for temporary StoreFiles. REVISION DETAIL https://reviews.facebook.net/D363 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockInfo.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaConfigured.java src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java Per-(table, columnFamily) metrics with configurable table name inclusion Key: HBASE-4768 URL: https://issues.apache.org/jira/browse/HBASE-4768 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D363.1.patch, D363.2.patch, D363.3.patch As we kept adding more granular block read and block cache usage statistics, a combinatorial explosion of various cases to monitor started to happen, especially when we wanted both per-table/column family/block type statistics and aggregate statistics on various subsets of these dimensions. Here, we un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a centralized class that knows how to update all kinds of per-table/CF/block type counters. Table name and column family configuration have been pushed to a base class, SchemaConfigured. This is convenient as many of existing classes that have these properties (HFile readers/writers, HFile blocks, etc.) did not have a base class. Whether to collect per-(table, columnFamily) or per-columnFamily only metrics can be configured with the hbase.metrics.showTableName configuration key. We don't expect this configuration to change at runtime, so we cache the setting statically and log a warning when an attempt is made to flip it once already set. This way we don't have to pass configuration to a lot more places, e.g. everywhere an HFile reader is instantiated. Thanks to Liyin for his initial version of per-table metrics patch and a lot of valuable feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on
[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion
[ https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149085#comment-13149085 ] Phabricator commented on HBASE-4768: tedyu has commented on the revision [jira] [HBASE-4768] Per-(table, columnFamily) metrics with configurable table name inclusion. Nice work. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:2 Year isn't needed. src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:43 The class should be SchemaConfigured. src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:37 Should read 'can be associated with table' src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:89 We shouldn't say HFile path because the path is too shallow, right ? src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:153 Since either tableName or cfName may not be null, should we include them in the exception message ? src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:166 Would passSchemaMetricsTo() be a better name for this method ? src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:170 I didn't find reference to this constant in the patch. From its format, it doesn't look like a prefix (no trailing .). REVISION DETAIL https://reviews.facebook.net/D363 Per-(table, columnFamily) metrics with configurable table name inclusion Key: HBASE-4768 URL: https://issues.apache.org/jira/browse/HBASE-4768 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D363.1.patch, D363.2.patch, D363.3.patch As we kept adding more granular block read and block cache usage statistics, a combinatorial explosion of various cases to monitor started to happen, especially when we wanted both per-table/column family/block type statistics and aggregate statistics on various subsets of these dimensions. Here, we un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a centralized class that knows how to update all kinds of per-table/CF/block type counters. Table name and column family configuration have been pushed to a base class, SchemaConfigured. This is convenient as many of existing classes that have these properties (HFile readers/writers, HFile blocks, etc.) did not have a base class. Whether to collect per-(table, columnFamily) or per-columnFamily only metrics can be configured with the hbase.metrics.showTableName configuration key. We don't expect this configuration to change at runtime, so we cache the setting statically and log a warning when an attempt is made to flip it once already set. This way we don't have to pass configuration to a lot more places, e.g. everywhere an HFile reader is instantiated. Thanks to Liyin for his initial version of per-table metrics patch and a lot of valuable feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4610) Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug)
[ https://issues.apache.org/jira/browse/HBASE-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149094#comment-13149094 ] Ted Yu commented on HBASE-4610: --- Since HBASE-4749 has been integrated, is this still needed ? Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug) - Key: HBASE-4610 URL: https://issues.apache.org/jira/browse/HBASE-4610 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Fix For: 0.92.0 Over in HBASE-3380 we were having some TestMasterFailover flakiness. We added some more config parameters to better control the master startup loop where it waits for RS to heartbeat in. We had thought at the time that 92 would have a different solution but it is still relying on heartbeats to learn about RSs. For now, we should definitely bring these config params into 92/trunk. Otherwise this is an incompatible regression and adding these will also make things like what was just reported over in HBASE-4603 trivial to fix in an optimal way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4575) Inconsistent naming for ZK config parameters
[ https://issues.apache.org/jira/browse/HBASE-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4575: -- Fix Version/s: (was: 0.92.0) 0.94.0 Inconsistent naming for ZK config parameters Key: HBASE-4575 URL: https://issues.apache.org/jira/browse/HBASE-4575 Project: HBase Issue Type: Bug Components: test, zookeeper Affects Versions: 0.92.0, 0.94.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Fix For: 0.94.0 I've found some misnaming of certain ZK config options. Make them consistent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4779) TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite
TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite - Key: HBASE-4779 URL: https://issues.apache.org/jira/browse/HBASE-4779 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor All tests must be tagged now. TestSuite is deprecated since JUnit 4.4, and does not works well with JUnit categories surefire. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4779) TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite
[ https://issues.apache.org/jira/browse/HBASE-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4779: --- Attachment: 4779_trunk_all.patch TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite - Key: HBASE-4779 URL: https://issues.apache.org/jira/browse/HBASE-4779 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4779_trunk_all.patch All tests must be tagged now. TestSuite is deprecated since JUnit 4.4, and does not works well with JUnit categories surefire. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4779) TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite
[ https://issues.apache.org/jira/browse/HBASE-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4779: --- Status: Patch Available (was: Open) TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite - Key: HBASE-4779 URL: https://issues.apache.org/jira/browse/HBASE-4779 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4779_trunk_all.patch All tests must be tagged now. TestSuite is deprecated since JUnit 4.4, and does not works well with JUnit categories surefire. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4779) TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite
[ https://issues.apache.org/jira/browse/HBASE-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149096#comment-13149096 ] Hadoop QA commented on HBASE-4779: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503495/4779_trunk_all.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/238//console This message is automatically generated. TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite - Key: HBASE-4779 URL: https://issues.apache.org/jira/browse/HBASE-4779 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4779_trunk_all.patch All tests must be tagged now. TestSuite is deprecated since JUnit 4.4, and does not works well with JUnit categories surefire. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest
[ https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149099#comment-13149099 ] nkeywal commented on HBASE-4764: @committers: please tell if you're ok with: - TestHLogUtils: renaming - SoftValueSortedMapTest: renaming + JUnitization if included tests works, deletion otherwise. naming errors for TestHLogUtils and SoftValueSortedMapTest -- Key: HBASE-4764 URL: https://issues.apache.org/jira/browse/HBASE-4764 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's not called. I don't know if it's used. TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses the script looking for the tests. It would seems a better thing to rename it. Is there anything special to do to keep the history attached to this file? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest
[ https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149102#comment-13149102 ] Ted Yu commented on HBASE-4764: --- I think the proposal makes sense. See also http://markphip.blogspot.com/2006/12/subversion-moverename-feature.html naming errors for TestHLogUtils and SoftValueSortedMapTest -- Key: HBASE-4764 URL: https://issues.apache.org/jira/browse/HBASE-4764 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's not called. I don't know if it's used. TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses the script looking for the tests. It would seems a better thing to rename it. Is there anything special to do to keep the history attached to this file? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4779) TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not tagged and TestPoolMap should not use TestSuite
[
https://issues.apache.org/jira/browse/HBASE-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149105#comment-13149105
]
Ted Yu commented on HBASE-4779:
---
Please fix the following:
{code}
Hunk #6 FAILED at 62.
3 out of 6 hunks FAILED -- saving rejects to file
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java.rej
{code}
TestHTablePool, TestScanWithBloomError, TestRegionSplitCalculator are not
tagged and TestPoolMap should not use TestSuite
-
Key: HBASE-4779
URL: https://issues.apache.org/jira/browse/HBASE-4779
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 4779_trunk_all.patch
All tests must be tagged now.
TestSuite is deprecated since JUnit 4.4, and does not works well with JUnit
categories surefire.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest
[ https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149110#comment-13149110 ] stack commented on HBASE-4764: -- +1 on proposed changes naming errors for TestHLogUtils and SoftValueSortedMapTest -- Key: HBASE-4764 URL: https://issues.apache.org/jira/browse/HBASE-4764 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's not called. I don't know if it's used. TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses the script looking for the tests. It would seems a better thing to rename it. Is there anything special to do to keep the history attached to this file? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4478) Improve AssignmentManager.handleRegion so that it can process certain ZK state in the case of RS offline
[
https://issues.apache.org/jira/browse/HBASE-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4478:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Improve AssignmentManager.handleRegion so that it can process certain ZK
state in the case of RS offline
Key: HBASE-4478
URL: https://issues.apache.org/jira/browse/HBASE-4478
Project: HBase
Issue Type: Bug
Reporter: Ming Ma
Assignee: ramkrishna.s.vasudevan
Attachments: 4478-v2.txt, HBASE-4478_1.patch
Currently AssignmentManager.handleRegion skips processing of ZK event change
if the RS is offline. It relies on TimeoutMonitor and ServerShutdownHandler
to process RIT.
// Verify this is a known server
if (!serverManager.isServerOnline(sn)
!this.master.getServerName().equals(sn)) {
LOG.warn(Attempted to handle region transition for server but +
server is not online: + Bytes.toString(data.getRegionName()));
return;
}
For certain states like OPENED, OPENING, FAILED_OPEN, CLOSED, it can continue
the progressing even if the RS is offline.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4777) Write back to client 'incompatible' if we show up with wrong version
[
https://issues.apache.org/jira/browse/HBASE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149195#comment-13149195
]
Ted Yu commented on HBASE-4777:
---
+1 on patch.
opentsdb would be very useful for HBase 0.92
Write back to client 'incompatible' if we show up with wrong version
Key: HBASE-4777
URL: https://issues.apache.org/jira/browse/HBASE-4777
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Attachments: 4777.txt
We changed the RPC_VERSION to 4 in hbase-3939. If a client comes in
volunteering RPC_VERSION is 3, currently, we'll log 'wrong version' but we'll
close the connection; the client has no chance of knowing why the server went
away.
Returning -1 as id up out of here is what causes the connection close:
{code}
private void setupBadVersionResponse(int clientVersion) throws
IOException {
String errMsg = Server IPC version + CURRENT_VERSION +
cannot communicate with client version + clientVersion;
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
if (clientVersion = 3) {
Call fakeCall = new Call(-1, null, this, responder);
// Versions 3 and greater can interpret this exception
// response in the same manner
setupResponse(buffer, fakeCall, Status.FATAL,
null, VersionMismatch.class.getName(), errMsg);
responder.doRespond(fakeCall);
}
}
{code}
Instead, we need to return an id that does not close the connection so cilent
gets chance to read the response.
Suggestion is that we return a 0 for the id the connection will stay up.
If an old client and it sends the wrong version, it'll move on to do
getProtocolVersion... and will fail there.
Other clients, e.g. asynchbase, if they get a response will have a response
to switch what they send to suit the new server.
(There are other issues -- e.g. Invocation is versioned now -- but Benoit
needs some means of figuring whats on other side)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4777) Write back to client 'incompatible' if we show up with wrong version
[
https://issues.apache.org/jira/browse/HBASE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149201#comment-13149201
]
Lars Hofhansl commented on HBASE-4777:
--
+1 as well
Write back to client 'incompatible' if we show up with wrong version
Key: HBASE-4777
URL: https://issues.apache.org/jira/browse/HBASE-4777
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Attachments: 4777.txt
We changed the RPC_VERSION to 4 in hbase-3939. If a client comes in
volunteering RPC_VERSION is 3, currently, we'll log 'wrong version' but we'll
close the connection; the client has no chance of knowing why the server went
away.
Returning -1 as id up out of here is what causes the connection close:
{code}
private void setupBadVersionResponse(int clientVersion) throws
IOException {
String errMsg = Server IPC version + CURRENT_VERSION +
cannot communicate with client version + clientVersion;
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
if (clientVersion = 3) {
Call fakeCall = new Call(-1, null, this, responder);
// Versions 3 and greater can interpret this exception
// response in the same manner
setupResponse(buffer, fakeCall, Status.FATAL,
null, VersionMismatch.class.getName(), errMsg);
responder.doRespond(fakeCall);
}
}
{code}
Instead, we need to return an id that does not close the connection so cilent
gets chance to read the response.
Suggestion is that we return a 0 for the id the connection will stay up.
If an old client and it sends the wrong version, it'll move on to do
getProtocolVersion... and will fail there.
Other clients, e.g. asynchbase, if they get a response will have a response
to switch what they send to suit the new server.
(There are other issues -- e.g. Invocation is versioned now -- but Benoit
needs some means of figuring whats on other side)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4777) Write back to client 'incompatible' if we show up with wrong version
[
https://issues.apache.org/jira/browse/HBASE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4777:
-
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for review lads. Committed branch and trunk.
Write back to client 'incompatible' if we show up with wrong version
Key: HBASE-4777
URL: https://issues.apache.org/jira/browse/HBASE-4777
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.92.0
Attachments: 4777.txt
We changed the RPC_VERSION to 4 in hbase-3939. If a client comes in
volunteering RPC_VERSION is 3, currently, we'll log 'wrong version' but we'll
close the connection; the client has no chance of knowing why the server went
away.
Returning -1 as id up out of here is what causes the connection close:
{code}
private void setupBadVersionResponse(int clientVersion) throws
IOException {
String errMsg = Server IPC version + CURRENT_VERSION +
cannot communicate with client version + clientVersion;
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
if (clientVersion = 3) {
Call fakeCall = new Call(-1, null, this, responder);
// Versions 3 and greater can interpret this exception
// response in the same manner
setupResponse(buffer, fakeCall, Status.FATAL,
null, VersionMismatch.class.getName(), errMsg);
responder.doRespond(fakeCall);
}
}
{code}
Instead, we need to return an id that does not close the connection so cilent
gets chance to read the response.
Suggestion is that we return a 0 for the id the connection will stay up.
If an old client and it sends the wrong version, it'll move on to do
getProtocolVersion... and will fail there.
Other clients, e.g. asynchbase, if they get a response will have a response
to switch what they send to suit the new server.
(There are other issues -- e.g. Invocation is versioned now -- but Benoit
needs some means of figuring whats on other side)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3025: - Priority: Critical (was: Major) Fix Version/s: 0.92.0 This is in way of 0.92 release. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025.2011-02-01.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149227#comment-13149227 ] jirapos...@reviews.apache.org commented on HBASE-3025: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2041/#review3187 --- About 1/4 way through. Will pick up again in morning. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java https://reviews.apache.org/r/2041/#comment7008 Is it on whenever I'm doing access control? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java https://reviews.apache.org/r/2041/#comment7009 Interesting security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7010 Is _acl_ a good name? I hate -ROOT- and then .META. Its dumb. All catalog tables should look the same. -ROOT- will likely go away soon. That would tend to rule the name be .ACL. But then leading off w/ a dot is a bit of a pain especially when you copy it local filesystem (it won't show in listings). On other hand, maybe thats ok... makes it special. And our *special* dirs tend to lead off with a '.' as in '.logs'., etc. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7011 We ensure a user can't have a '@ prefix I presume (haven't read all the code yet). security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7012 This is great. Nice. Clean. What happens if we ever want to do cell-level? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7013 I'd suggest you set an example with this new table and instead of having the cf be 'info', instead have it be 'l' as short for lists (you are giving an example by having short cf names). security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7014 If no family qualifier, we are granting perm on whole table? Thats ok? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7015 Should we check family is not null? Doesn't a qualifier have to have a family to qualify? This should be inside the family check? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7016 Ok good. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7017 Usually space either side of operators as in 'int i = 0' rather than 'int i=0'. Etc. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7018 Oh, a byte per action? Thats grand. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7019 Should it throw exception? Should we read for an ACL first and not write a delete if none present (throwing exception if nothing to delete)? I think I know why no effect security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7020 When I do this? And what happens if perms are edited subsequently? Are they considered? Or is this method for testing like the one that follows? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7021 We're doing the ACL table, not .META. This a stale comment? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7022 Stale comment? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7023 Just point at class comment rather than dup it here? security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7024 Probably have to do that over in core master classes. Its there we are guaranteeing root up before meta... security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java https://reviews.apache.org/r/2041/#comment7025 This is a bit of a pain having to make a String of it then going back to byte arrays after parse.
[jira] [Updated] (HBASE-2742) Provide strong authentication with a secure RPC engine
[ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2742: - Priority: Critical (was: Major) Fix Version/s: 0.92.0 This is in way of a 0.92 release. Provide strong authentication with a secure RPC engine -- Key: HBASE-2742 URL: https://issues.apache.org/jira/browse/HBASE-2742 Project: HBase Issue Type: Improvement Components: ipc Reporter: Gary Helmling Priority: Critical Fix For: 0.92.0 The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off of Hadoop RPC classes, with some performance tweaks added. Those optimizations have come at a cost in keeping up with Hadoop RPC changes however, both bug fixes and improvements/new features. In particular, this impacts how we implement security features in HBase (see HBASE-1697 and HBASE-2016). The secure Hadoop implementation (HADOOP-4487) relies heavily on RPC changes to support client authentication via kerberos and securing and mutual authentication of client/server connections via SASL. Making use of the built-in Hadoop RPC classes will gain us these pieces for free in a secure HBase. So, I'm proposing that we drop the HBase forked version of RPC and convert to direct use of Hadoop RPC, while working to contribute important fixes back upstream to Hadoop core. Based on a review of the HBase RPC changes, the key divergences seem to be: HBaseClient: - added use of TCP keepalive (HBASE-1754) - made connection retries and sleep configurable (HBASE-1815) - prevent NPE if socket == null due to creation failure (HBASE-2443) HBaseRPC: - mapping of method names - codes (removed in HBASE-2219) HBaseServer: - use of TCP keep alives (HBASE-1754) - OOME in server does not trigger abort (HBASE-1198) HbaseObjectWritable: - allows List serialization - includes it's own class - code mapping (HBASE-328) Proposed process is: 1. open issues with patches on Hadoop core for important fixes/adjustments from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a pluggable ObjectWritable implementation in RPC.Invocation to allow use of HbaseObjectWritable). 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid another copy-n-paste code fork, subject to ability to isolate changes from impacting Hadoop internal RPC wire formats 3. if all Hadoop core patches are applied we can drop back to a plain vanilla Hadoop version I realize there are many different opinions on how to proceed with HBase RPC, so I'm hoping this issue will kick off a discussion on what the best approach might be. My own motivation is maximizing re-use of the authentication and connection security work that's already gone into Hadoop core. I'll put together a set of patches around #1 and #2, but obviously we need some consensus around this to move forward. If I'm missing other differences between HBase and Hadoop RPC, please list as well. Discuss! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion
[ https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149232#comment-13149232 ] Phabricator commented on HBASE-4768: mbautin has commented on the revision [jira] [HBASE-4768] Per-(table, columnFamily) metrics with configurable table name inclusion. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:2 FYI, no year in copyright message is the least frequent option in the current codebase, but since it doesn't really matter, I can remove the year. 854 Copyright 2010 The Apache Software Foundation 360 Copyright 2011 The Apache Software Foundation 158 Copyright 2009 The Apache Software Foundation 108 Copyright 2007 The Apache Software Foundation 40 Copyright 2008 The Apache Software Foundation 6 Copyright The Apache Software Foundation src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:37 Fixed. src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:43 Fixed. src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java:89 Replaced with: LOG.warn(Could not determine table and column family of the HFile + path + path + . Expecting at least + HFile.MIN_NUM_HFILE_PATH_LEVELS + path components.); src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java:170 Good catch. These are actually used in some metrics in hbase-89 that are not yet ported to trunk. Removing them from this patch. REVISION DETAIL https://reviews.facebook.net/D363 Per-(table, columnFamily) metrics with configurable table name inclusion Key: HBASE-4768 URL: https://issues.apache.org/jira/browse/HBASE-4768 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D363.1.patch, D363.2.patch, D363.3.patch As we kept adding more granular block read and block cache usage statistics, a combinatorial explosion of various cases to monitor started to happen, especially when we wanted both per-table/column family/block type statistics and aggregate statistics on various subsets of these dimensions. Here, we un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a centralized class that knows how to update all kinds of per-table/CF/block type counters. Table name and column family configuration have been pushed to a base class, SchemaConfigured. This is convenient as many of existing classes that have these properties (HFile readers/writers, HFile blocks, etc.) did not have a base class. Whether to collect per-(table, columnFamily) or per-columnFamily only metrics can be configured with the hbase.metrics.showTableName configuration key. We don't expect this configuration to change at runtime, so we cache the setting statically and log a warning when an attempt is made to flip it once already set. This way we don't have to pass configuration to a lot more places, e.g. everywhere an HFile reader is instantiated. Thanks to Liyin for his initial version of per-table metrics patch and a lot of valuable feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149234#comment-13149234 ] Andrew Purtell commented on HBASE-3025: --- I'll sort out the disposition of this and the other two patches with Gary next week. With any luck we'll get them committed next week as well. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025.2011-02-01.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.
[
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suraj Varma updated HBASE-4565:
---
Attachment: HBASE-4565-v3-0.92.patch
HBASE-4565-v3.patch
The tarball in the previous patches had an extra directory. Attaching v3
versions for both trunk and 0.92
Maven HBase build broken on cygwin with copynativelib.sh call.
--
Key: HBASE-4565
URL: https://issues.apache.org/jira/browse/HBASE-4565
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.92.0
Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
Labels: build, maven
Fix For: 0.94.0
Attachments: HBASE-4565-0.92.patch, HBASE-4565-v2.patch,
HBASE-4565.patch
This is broken in both 0.92 as well as trunk pom.xml
Here's a sample maven log snippet from trunk (from Mayuresh on user mailing
list)
[INFO] [antrun:run {execution: package}]
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
[exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib:
No such file or directory
[exec] tar (child): Cannot connect to D: resolve failed
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] An Ant BuildException has occured: exec returned: 3328
There are two issues:
1) The ant run task below doesn't resolve the windows file separator returned
by the project.build.directory - this causes the above resolve failed.
!-- Using Unix cp to preserve symlinks, using script to handle wildcards --
echo file=${project.build.directory}/copynativelibs.sh
if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
2) The tar argument value below also has a similar issue in that the path arg
doesn't resolve right.
!-- Using Unix tar to preserve symlinks --
exec executable=tar failonerror=yes
dir=${project.build.directory}/${project.artifactId}-${project.version}
arg value=czf/
arg
value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
arg value=./
/exec
In both cases, the fix would probably be to use a cross-platform way to
handle the directory locations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.
[
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suraj Varma updated HBASE-4565:
---
Attachment: (was: HBASE-4565-v3.patch)
Maven HBase build broken on cygwin with copynativelib.sh call.
--
Key: HBASE-4565
URL: https://issues.apache.org/jira/browse/HBASE-4565
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.92.0
Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
Labels: build, maven
Fix For: 0.94.0
Attachments: HBASE-4565-0.92.patch, HBASE-4565-v2.patch,
HBASE-4565.patch
This is broken in both 0.92 as well as trunk pom.xml
Here's a sample maven log snippet from trunk (from Mayuresh on user mailing
list)
[INFO] [antrun:run {execution: package}]
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
[exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib:
No such file or directory
[exec] tar (child): Cannot connect to D: resolve failed
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] An Ant BuildException has occured: exec returned: 3328
There are two issues:
1) The ant run task below doesn't resolve the windows file separator returned
by the project.build.directory - this causes the above resolve failed.
!-- Using Unix cp to preserve symlinks, using script to handle wildcards --
echo file=${project.build.directory}/copynativelibs.sh
if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
2) The tar argument value below also has a similar issue in that the path arg
doesn't resolve right.
!-- Using Unix tar to preserve symlinks --
exec executable=tar failonerror=yes
dir=${project.build.directory}/${project.artifactId}-${project.version}
arg value=czf/
arg
value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
arg value=./
/exec
In both cases, the fix would probably be to use a cross-platform way to
handle the directory locations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.
[
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suraj Varma updated HBASE-4565:
---
Attachment: (was: HBASE-4565-v3-0.92.patch)
Maven HBase build broken on cygwin with copynativelib.sh call.
--
Key: HBASE-4565
URL: https://issues.apache.org/jira/browse/HBASE-4565
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.92.0
Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
Labels: build, maven
Fix For: 0.94.0
Attachments: HBASE-4565-0.92.patch, HBASE-4565-v2.patch,
HBASE-4565.patch
This is broken in both 0.92 as well as trunk pom.xml
Here's a sample maven log snippet from trunk (from Mayuresh on user mailing
list)
[INFO] [antrun:run {execution: package}]
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
[exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib:
No such file or directory
[exec] tar (child): Cannot connect to D: resolve failed
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] An Ant BuildException has occured: exec returned: 3328
There are two issues:
1) The ant run task below doesn't resolve the windows file separator returned
by the project.build.directory - this causes the above resolve failed.
!-- Using Unix cp to preserve symlinks, using script to handle wildcards --
echo file=${project.build.directory}/copynativelibs.sh
if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
2) The tar argument value below also has a similar issue in that the path arg
doesn't resolve right.
!-- Using Unix tar to preserve symlinks --
exec executable=tar failonerror=yes
dir=${project.build.directory}/${project.artifactId}-${project.version}
arg value=czf/
arg
value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
arg value=./
/exec
In both cases, the fix would probably be to use a cross-platform way to
handle the directory locations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.
[
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suraj Varma updated HBASE-4565:
---
Attachment: HBASE-4565-v3.patch
HBASE-4565-v3-0.92.patch
Maven HBase build broken on cygwin with copynativelib.sh call.
--
Key: HBASE-4565
URL: https://issues.apache.org/jira/browse/HBASE-4565
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.92.0
Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
Labels: build, maven
Fix For: 0.94.0
Attachments: HBASE-4565-0.92.patch, HBASE-4565-v2.patch,
HBASE-4565-v3-0.92.patch, HBASE-4565-v3.patch, HBASE-4565.patch
This is broken in both 0.92 as well as trunk pom.xml
Here's a sample maven log snippet from trunk (from Mayuresh on user mailing
list)
[INFO] [antrun:run {execution: package}]
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
[exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib:
No such file or directory
[exec] tar (child): Cannot connect to D: resolve failed
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] An Ant BuildException has occured: exec returned: 3328
There are two issues:
1) The ant run task below doesn't resolve the windows file separator returned
by the project.build.directory - this causes the above resolve failed.
!-- Using Unix cp to preserve symlinks, using script to handle wildcards --
echo file=${project.build.directory}/copynativelibs.sh
if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
2) The tar argument value below also has a similar issue in that the path arg
doesn't resolve right.
!-- Using Unix tar to preserve symlinks --
exec executable=tar failonerror=yes
dir=${project.build.directory}/${project.artifactId}-${project.version}
arg value=czf/
arg
value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
arg value=./
/exec
In both cases, the fix would probably be to use a cross-platform way to
handle the directory locations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.
[
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suraj Varma updated HBASE-4565:
---
Status: Patch Available (was: Open)
Maven HBase build broken on cygwin with copynativelib.sh call.
--
Key: HBASE-4565
URL: https://issues.apache.org/jira/browse/HBASE-4565
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.92.0
Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
Labels: build, maven
Fix For: 0.94.0
Attachments: HBASE-4565-0.92.patch, HBASE-4565-v2.patch,
HBASE-4565-v3-0.92.patch, HBASE-4565-v3.patch, HBASE-4565.patch
This is broken in both 0.92 as well as trunk pom.xml
Here's a sample maven log snippet from trunk (from Mayuresh on user mailing
list)
[INFO] [antrun:run {execution: package}]
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
[exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib:
No such file or directory
[exec] tar (child): Cannot connect to D: resolve failed
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] An Ant BuildException has occured: exec returned: 3328
There are two issues:
1) The ant run task below doesn't resolve the windows file separator returned
by the project.build.directory - this causes the above resolve failed.
!-- Using Unix cp to preserve symlinks, using script to handle wildcards --
echo file=${project.build.directory}/copynativelibs.sh
if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
2) The tar argument value below also has a similar issue in that the path arg
doesn't resolve right.
!-- Using Unix tar to preserve symlinks --
exec executable=tar failonerror=yes
dir=${project.build.directory}/${project.artifactId}-${project.version}
arg value=czf/
arg
value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
arg value=./
/exec
In both cases, the fix would probably be to use a cross-platform way to
handle the directory locations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4777) Write back to client 'incompatible' if we show up with wrong version
[
https://issues.apache.org/jira/browse/HBASE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149235#comment-13149235
]
Hudson commented on HBASE-4777:
---
Integrated in HBase-TRUNK #2432 (See
[https://builds.apache.org/job/HBase-TRUNK/2432/])
HBASE-4777 Write back to client 'incompatible' if we show up with wrong
version
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
Write back to client 'incompatible' if we show up with wrong version
Key: HBASE-4777
URL: https://issues.apache.org/jira/browse/HBASE-4777
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.92.0
Attachments: 4777.txt
We changed the RPC_VERSION to 4 in hbase-3939. If a client comes in
volunteering RPC_VERSION is 3, currently, we'll log 'wrong version' but we'll
close the connection; the client has no chance of knowing why the server went
away.
Returning -1 as id up out of here is what causes the connection close:
{code}
private void setupBadVersionResponse(int clientVersion) throws
IOException {
String errMsg = Server IPC version + CURRENT_VERSION +
cannot communicate with client version + clientVersion;
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
if (clientVersion = 3) {
Call fakeCall = new Call(-1, null, this, responder);
// Versions 3 and greater can interpret this exception
// response in the same manner
setupResponse(buffer, fakeCall, Status.FATAL,
null, VersionMismatch.class.getName(), errMsg);
responder.doRespond(fakeCall);
}
}
{code}
Instead, we need to return an id that does not close the connection so cilent
gets chance to read the response.
Suggestion is that we return a 0 for the id the connection will stay up.
If an old client and it sends the wrong version, it'll move on to do
getProtocolVersion... and will fail there.
Other clients, e.g. asynchbase, if they get a response will have a response
to switch what they send to suit the new server.
(There are other issues -- e.g. Invocation is versioned now -- but Benoit
needs some means of figuring whats on other side)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
[
https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149242#comment-13149242
]
jirapos...@reviews.apache.org commented on HBASE-2742:
--
bq. On 2011-11-04 23:17:46, Michael Stack wrote:
bq. pom.xml, line 1335
bq. https://reviews.apache.org/r/1991/diff/4/?file=53576#file53576line1335
bq.
bq. This is probably best place for this code for now. Was thinking
src/security but that gets weird when test code and main. This is
pseudo-maven-modules for now.
Yeah, my thinking was once we get to full blown modules we won't have to move
the source around, just add a security pom.xml.
bq. On 2011-11-04 23:17:46, Michael Stack wrote:
bq. conf/hbase-policy.xml, line 1
bq. https://reviews.apache.org/r/1991/diff/4/?file=53575#file53575line1
bq.
bq. Apache license?
Added.
bq. On 2011-11-04 23:17:46, Michael Stack wrote:
bq. pom.xml, line 1320
bq. https://reviews.apache.org/r/1991/diff/4/?file=53576#file53576line1320
bq.
bq. We need to change this when we ship 0.92? Can you use variable
here? ${pom.version}?
Removed this bit to use the project version. I'm trying out using finalName in
the profile, which would allow us to wind up with the name
hbase-0.92.0-security. Dependent projects could then use artifactId=hbase,
version=0.92.0, classifier=security. Or we could use versions:set
-DnewVersion=0.92.0-security during the build, but making it more automatic
seems nicer.
bq. On 2011-11-04 23:17:46, Michael Stack wrote:
bq. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java,
line 158
bq. https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line158
bq.
bq. Minor nit: My guess is that this is not your code. Why not just a
return CONDITIONS... no need of this if ... return true else return false
Yes, was copied. Changed to just do a return.
- Gary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1991/#review3058
---
On 2011-10-26 20:23:19, Gary Helmling wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1991/
bq. ---
bq.
bq. (Updated 2011-10-26 20:23:19)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This patch creates a new secure RPC engine for HBase, which provides
Kerberos based authentication of clients, and a token-based authentication
mechanism for mapreduce jobs. Primary components of the patch are:
bq.
bq. - a new maven profile for secure Hadoop/HBase: hadoop-0.20S
bq.- Secure Hadoop dependent classes are separated under a pseudo-module in
the security/ directory. These source and test directories are only including
if building the secure Hadoop profile
bq.- Currently the security classes get packaged with the regular HBase
build artifacts. We need a way to at least override project.version, so we can
append something like a -security suffix indicating the additional security
components.
bq.- The pseudo-module here is really a half-step forward. It enables the
security code to be optionally included in the build for now, and sets up the
structure for a security module. But we still will want to pursue full
modularization (see HBASE-4336), which will allow packing the security code in
a separate build artifact.
bq.
bq. - a new RPC engine providing kerberos and token-based authentication:
org.apache.hadoop.hbase.ipc.SecureRpcEngine
bq.- implementation under
security/src/main/java/org/apache/hadoop/hbase/ipc/
bq.- The implementation classes extend the existing HBaseClient and
HBaseServer to share as much of the RPC code as possible. The main override is
of the connection classes to allow control over the SASL negotiation of secure
connections
bq.
bq. - existing RPC changes
bq.- The existing HBaseClient and HBaseServer have been modified to make
subclassing possible
bq.- All references to Hadoop UserGroupInformation have been replaced with
org.apache.hadoop.hbase.security.User to insulate from future dependencies on
specific Hadoop versions
bq.
bq. - a coprocessor endpoint for obtaining new authentication tokens:
TokenProvider, and supporting classes for token generation and synchronization
(incorporating HBASE-3615)
bq.- implementation is under
security/src/main/java/org/apache/hadoop/hbase/security/token/
bq.- Secret keys for token generation and verification are synchronized
throughout the cluster in zookeeper, under /hbase/tokenauth/keys
bq.
bq.
bq. To enable secure RPC, add the following configuration
[jira] [Commented] (HBASE-4777) Write back to client 'incompatible' if we show up with wrong version
[
https://issues.apache.org/jira/browse/HBASE-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149244#comment-13149244
]
Hudson commented on HBASE-4777:
---
Integrated in HBase-0.92 #128 (See
[https://builds.apache.org/job/HBase-0.92/128/])
HBASE-4777 Write back to client 'incompatible' if we show up with wrong
version
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
Write back to client 'incompatible' if we show up with wrong version
Key: HBASE-4777
URL: https://issues.apache.org/jira/browse/HBASE-4777
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.92.0
Attachments: 4777.txt
We changed the RPC_VERSION to 4 in hbase-3939. If a client comes in
volunteering RPC_VERSION is 3, currently, we'll log 'wrong version' but we'll
close the connection; the client has no chance of knowing why the server went
away.
Returning -1 as id up out of here is what causes the connection close:
{code}
private void setupBadVersionResponse(int clientVersion) throws
IOException {
String errMsg = Server IPC version + CURRENT_VERSION +
cannot communicate with client version + clientVersion;
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
if (clientVersion = 3) {
Call fakeCall = new Call(-1, null, this, responder);
// Versions 3 and greater can interpret this exception
// response in the same manner
setupResponse(buffer, fakeCall, Status.FATAL,
null, VersionMismatch.class.getName(), errMsg);
responder.doRespond(fakeCall);
}
}
{code}
Instead, we need to return an id that does not close the connection so cilent
gets chance to read the response.
Suggestion is that we return a 0 for the id the connection will stay up.
If an old client and it sends the wrong version, it'll move on to do
getProtocolVersion... and will fail there.
Other clients, e.g. asynchbase, if they get a response will have a response
to switch what they send to suit the new server.
(There are other issues -- e.g. Invocation is versioned now -- but Benoit
needs some means of figuring whats on other side)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion
[ https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-4768: --- Attachment: D363.4.patch mbautin updated the revision [jira] [HBASE-4768] Per-(table, columnFamily) metrics with configurable table name inclusion. Reviewers: jgray, nspiegelberg, stack, tedyu, todd, JIRA Addressing Ted's comments. All unit tests passed. REVISION DETAIL https://reviews.facebook.net/D363 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockInfo.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaConfigured.java src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java Per-(table, columnFamily) metrics with configurable table name inclusion Key: HBASE-4768 URL: https://issues.apache.org/jira/browse/HBASE-4768 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D363.1.patch, D363.2.patch, D363.3.patch, D363.4.patch As we kept adding more granular block read and block cache usage statistics, a combinatorial explosion of various cases to monitor started to happen, especially when we wanted both per-table/column family/block type statistics and aggregate statistics on various subsets of these dimensions. Here, we un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a centralized class that knows how to update all kinds of per-table/CF/block type counters. Table name and column family configuration have been pushed to a base class, SchemaConfigured. This is convenient as many of existing classes that have these properties (HFile readers/writers, HFile blocks, etc.) did not have a base class. Whether to collect per-(table, columnFamily) or per-columnFamily only metrics can be configured with the hbase.metrics.showTableName configuration key. We don't expect this configuration to change at runtime, so we cache the setting statically and log a warning when an attempt is made to flip it once already set. This way we don't have to pass configuration to a lot more places, e.g. everywhere an HFile reader is instantiated. Thanks to Liyin for his initial version of per-table metrics patch and a lot of valuable feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
