[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256735#comment-13256735 ] Phabricator commented on HBASE-5813: mbautin has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1736 We add locations for all regions participating in the query to newRegionLocations in the initial loop of this function. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1663 Good catch! REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v8 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256768#comment-13256768 ] Phabricator commented on HBASE-5813: tedyu has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. I cannot accept a second time :-) REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v14 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch, D2847.6.patch, D2847.7.patch, D2847.8.patch, D2847.9.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256900#comment-13256900 ] Phabricator commented on HBASE-5813: khemani has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1733 It is probably better to wrap the InterruptedException into a InterruptedIoException and throw it immediately. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1665 Is a TreeMap needed? Or a HashMap would have worked? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1271 similar logic ought to be present here in getRegionServerWithRetries() REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v14 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch, D2847.6.patch, D2847.7.patch, D2847.8.patch, D2847.9.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256904#comment-13256904 ] Phabricator commented on HBASE-5813: aaiyer has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. Looks good. REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v14 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch, D2847.6.patch, D2847.7.patch, D2847.8.patch, D2847.9.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256947#comment-13256947 ] Phabricator commented on HBASE-5813: mbautin has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1665 HashMap cannot be keyed by byte[] (see previous comments on this diff) REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v16 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.10.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch, D2847.6.patch, D2847.7.patch, D2847.8.patch, D2847.9.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256986#comment-13256986 ] Phabricator commented on HBASE-5813: khemani has requested changes to the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. Requesting Changes so that it can get accepted again REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v17 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.10.patch, D2847.11.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch, D2847.6.patch, D2847.7.patch, D2847.8.patch, D2847.9.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256988#comment-13256988 ] Phabricator commented on HBASE-5813: mbautin has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. All unit tests passed. I also ran Test*ServerCmdLine tests from D2811 and checked the logs for retries after a NotServingRegionException. This fix did save two five-second pauses in those tests. REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v17 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.10.patch, D2847.11.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch, D2847.6.patch, D2847.7.patch, D2847.8.patch, D2847.9.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256990#comment-13256990 ] Phabricator commented on HBASE-5813: khemani has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. The ideal behavior will be the following look up cached location retry: make the call if the call fails with NSRE look up cache for location with reload equal to true if you get the same location again then wait lookup cache for location with reload equal to not sure what goto retry else goto retry The difference is that you are not making an extra roundtrip to the region server even when you have discovered that prevAddress is same as newAddress. REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v17 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.10.patch, D2847.11.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch, D2847.6.patch, D2847.7.patch, D2847.8.patch, D2847.9.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255978#comment-13255978 ] Phabricator commented on HBASE-5813: tedyu has accepted the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1629 Please remove 'after' or change it to 'after failure'. REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v2 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.2.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3508) LruBlockCache statistics thread should have a name
[ https://issues.apache.org/jira/browse/HBASE-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256004#comment-13256004 ] Phabricator commented on HBASE-3508: Liyin has accepted the revision [jira] [HBASE-3508] [89-fb] LruBlockCache statistics thread should have a name. LGTM:) One minor comment. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java:129 May set it as a daemon thread here REVISION DETAIL https://reviews.facebook.net/D2853 BRANCH lrublockcache_statistics_thread_should_have_a_HBASE-3508 LruBlockCache statistics thread should have a name -- Key: HBASE-3508 URL: https://issues.apache.org/jira/browse/HBASE-3508 Project: HBase Issue Type: Improvement Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial Fix For: 0.90.1 Attachments: D2853.1.patch, hbase-3508.txt Currently it's just an unnamed threadpool. It's annoying. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5803) [89-fb] Upgrade hbase 0.89-fb to Thrift 0.8.0 and bring Thrift server enhancements from trunk
[ https://issues.apache.org/jira/browse/HBASE-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256040#comment-13256040 ] Phabricator commented on HBASE-5803: sc has accepted the revision [jira] [HBASE-5803] [89-fb] Upgrade hbase 0.89-fb to Thrift 0.8.0 and bring Thrift server enhancements from trunk. Looks good to me. Good work! REVISION DETAIL https://reviews.facebook.net/D2811 BRANCH upgrade_hbase_0_89_fb_to_thrift_0_8_0_and_HBASE-5803_v9 [89-fb] Upgrade hbase 0.89-fb to Thrift 0.8.0 and bring Thrift server enhancements from trunk - Key: HBASE-5803 URL: https://issues.apache.org/jira/browse/HBASE-5803 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2811.1.patch, D2811.2.patch, D2811.3.patch TBoundedThreadPoolServer has been a problem for us when there is a large number of clients. We need to migrate to 0.8.0. in 89-fb and bring the relevant improvements from trunk, including supporting TThreadedSelectorServer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256100#comment-13256100 ] Phabricator commented on HBASE-5763: mbautin has commented on the revision [jira] [HBASE-5763] Fix random failures in TestFSErrorsExposed. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java:535 Done. REVISION DETAIL https://reviews.facebook.net/D2793 BRANCH fix_random_failures_in_testfserrorsexposed_HBASE-5763_v5 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch, D2739.3.patch, D2739.4.patch, D2793.1.patch, D2793.2.patch, D2793.3.patch, Fix-TestFSErrorsExposed-2012-04-13_18_59_36.patch, Fix-TestFSErrorsExposed-2012-04-16_15_41_24.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256105#comment-13256105 ] Phabricator commented on HBASE-5763: mbautin has committed the revision [jira] [HBASE-5763] Fix random failures in TestFSErrorsExposed. REVISION DETAIL https://reviews.facebook.net/D2793 COMMIT https://reviews.facebook.net/rHBASE1327337 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch, D2739.3.patch, D2739.4.patch, D2793.1.patch, D2793.2.patch, D2793.3.patch, Fix-TestFSErrorsExposed-2012-04-13_18_59_36.patch, Fix-TestFSErrorsExposed-2012-04-16_15_41_24.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256172#comment-13256172 ] Phabricator commented on HBASE-5813: tedyu has commented on the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1736 newRegionLocations.get(regionName) may return null, right ? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1663 Should 'new TreeMapbyte[], HServerAddress(Bytes.BYTES_COMPARATOR)' be used here ? REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v8 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5813) Retry immediately after a NotServingRegionException in a multiput
[ https://issues.apache.org/jira/browse/HBASE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256205#comment-13256205 ] Phabricator commented on HBASE-5813: Kannan has added reviewers to the revision [jira] [HBASE-5813] [89-fb] Retry immediately after a NotServingRegionException in a multiput. Added Reviewers: khemani, aaiyer REVISION DETAIL https://reviews.facebook.net/D2847 BRANCH retry_immediately_after_a_HBASE-5813_v8 Retry immediately after a NotServingRegionException in a multiput - Key: HBASE-5813 URL: https://issues.apache.org/jira/browse/HBASE-5813 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2847.1.patch, D2847.2.patch, D2847.3.patch, D2847.4.patch, D2847.5.patch After we get some errors in a multiput we invalidate the region location cache and wait for the configured time interval according to the backoff policy. However, if all errors in multiput processing were NotServingRegionExceptions, we don't really need to wait. We can retry immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255147#comment-13255147 ] Phabricator commented on HBASE-5763: tedyu has accepted the revision [jira] [HBASE-5763] Fix random failures in TestFSErrorsExposed. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java:535 It would be better if javadoc is added for the new public methods. REVISION DETAIL https://reviews.facebook.net/D2793 BRANCH fix_random_failures_in_testfserrorsexposed_HBASE-5763_v3 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch, D2739.3.patch, D2739.4.patch, D2793.1.patch, D2793.2.patch, Fix-TestFSErrorsExposed-2012-04-13_18_59_36.patch, Fix-TestFSErrorsExposed-2012-04-16_15_41_24.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5803) [89-fb] Upgrade hbase 0.89-fb to Thrift 0.8.0 and bring Thrift server enhancements from trunk
[ https://issues.apache.org/jira/browse/HBASE-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255305#comment-13255305 ] Phabricator commented on HBASE-5803: sc has commented on the revision [jira] [HBASE-5803] [89-fb] Upgrade hbase 0.89-fb to Thrift 0.8.0 and bring Thrift server enhancements from trunk. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:888 There are some places that we use ByteBuffer.array() instead of reading from the correct offsets. They cause problem when using framed transport protocols. See https://issues.apache.org/jira/browse/HBASE-5736 REVISION DETAIL https://reviews.facebook.net/D2811 [89-fb] Upgrade hbase 0.89-fb to Thrift 0.8.0 and bring Thrift server enhancements from trunk - Key: HBASE-5803 URL: https://issues.apache.org/jira/browse/HBASE-5803 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2811.1.patch TBoundedThreadPoolServer has been a problem for us when there is a large number of clients. We need to migrate to 0.8.0. in 89-fb and bring the relevant improvements from trunk, including supporting TThreadedSelectorServer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5104) Provide a reliable intra-row pagination mechanism
[ https://issues.apache.org/jira/browse/HBASE-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254498#comment-13254498 ] Phabricator commented on HBASE-5104: mbautin has commented on the revision [jira] [HBASE-5104] Provide a reliable intra-row pagination mechanism. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/Get.java:470 Removed. src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java:395 Done. src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide2.java:38 Done. REVISION DETAIL https://reviews.facebook.net/D2799 Provide a reliable intra-row pagination mechanism - Key: HBASE-5104 URL: https://issues.apache.org/jira/browse/HBASE-5104 Project: HBase Issue Type: Bug Reporter: Kannan Muthukkaruppan Assignee: Madhuwanti Vaidya Attachments: D2799.1.patch, D2799.2.patch, testFilterList.rb Addendum: Doing pagination (retrieving at most limit number of KVs at a particular offset) is currently supported via the ColumnPaginationFilter. However, it is not a very clean way of supporting pagination. Some of the problems with it are: * Normally, one would expect a query with (Filter(A) AND Filter(B)) to have same results as (query with Filter(A)) INTERSECT (query with Filter(B)). This is not the case for ColumnPaginationFilter as its internal state gets updated depending on whether or not Filter(A) returns TRUE/FALSE for a particular cell. * When this Filter is used in combination with other filters (e.g., doing AND with another filter using FilterList), the behavior of the query depends on the order of filters in the FilterList. This is not ideal. * ColumnPaginationFilter is a stateful filter which ends up counting multiple versions of the cell as separate values even if another filter upstream or the ScanQueryMatcher is going to reject the value for other reasons. Seems like we need a reliable way to do pagination. The particular use case that prompted this JIRA is pagination within the same rowKey. For example, for a given row key R, get columns with prefix P, starting at offset X (among columns which have prefix P) and limit Y. Some possible fixes might be: 1) enhance ColumnPrefixFilter to support another constructor which supports limit/offset. 2) Support pagination (limit/offset) at the Scan/Get API level (rather than as a filter) [Like SQL]. Original Post: Thanks Jiakai Liu for reporting this issue and doing the initial investigation. Email from Jiakai below: Assuming that we have an index column family with the following entries: tag0:001:thread1 ... tag1:001:thread1 tag1:002:thread2 ... tag1:010:thread10 ... tag2:001:thread1 tag2:005:thread5 ... To get threads with tag1 in range [5, 10), I tried the following code: ColumnPrefixFilter filter1 = new ColumnPrefixFilter(Bytes.toBytes(tag1)); ColumnPaginationFilter filter2 = new ColumnPaginationFilter(5 /* limit */, 5 /* offset */); FilterList filters = new FilterList(Operator.MUST_PASS_ALL); filters.addFilter(filter1); filters.addFilter(filter2); Get get = new Get(USER); get.addFamily(COLUMN_FAMILY); get.setMaxVersions(1); get.setFilter(filters); Somehow it didn't work as expected. It returned the entries as if the filter1 were not set. Turns out the ColumnPrefixFilter returns SEEK_NEXT_USING_HINT in some cases. The FilterList filter does not handle this return code properly (treat it as INCLUDE). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253926#comment-13253926 ] Phabricator commented on HBASE-5684: mbautin has committed the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. REVISION DETAIL https://reviews.facebook.net/D2757 COMMIT https://reviews.facebook.net/rHBASE1326036 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253957#comment-13253957 ] Phabricator commented on HBASE-5684: mbautin has committed the revision [jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. REVISION DETAIL https://reviews.facebook.net/D2709 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1326045 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch, D2757.2.patch, D2757.3.patch, D2757.4.patch, jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253958#comment-13253958 ] Phabricator commented on HBASE-5763: mbautin has committed the revision [jira] [HBASE-5763] [89-fb] Fix random failures in TestFSErrorsExposed. REVISION DETAIL https://reviews.facebook.net/D2739 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1326046 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch, D2739.3.patch, D2739.4.patch, D2793.1.patch, Fix-TestFSErrorsExposed-2012-04-13_18_59_36.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253977#comment-13253977 ] Phabricator commented on HBASE-5763: stack has commented on the revision [jira] [HBASE-5763] [89-fb] Fix random failures in TestFSErrorsExposed. lgtm REVISION DETAIL https://reviews.facebook.net/D2739 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1326046 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch, D2739.3.patch, D2739.4.patch, D2793.1.patch, Fix-TestFSErrorsExposed-2012-04-13_18_59_36.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5104) Provide a reliable intra-row pagination mechanism
[ https://issues.apache.org/jira/browse/HBASE-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253981#comment-13253981 ] Phabricator commented on HBASE-5104: tedyu has commented on the revision [jira] [HBASE-5104] Provide a reliable intra-row pagination mechanism. Nice work. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/Get.java:470 The assignment isn't needed here, right ? src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java:395 rowOffset - storeOffset src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide2.java:38 Add test category, please. REVISION DETAIL https://reviews.facebook.net/D2799 Provide a reliable intra-row pagination mechanism - Key: HBASE-5104 URL: https://issues.apache.org/jira/browse/HBASE-5104 Project: HBase Issue Type: Bug Reporter: Kannan Muthukkaruppan Assignee: Madhuwanti Vaidya Attachments: D2799.1.patch, D2799.2.patch, testFilterList.rb Addendum: Doing pagination (retrieving at most limit number of KVs at a particular offset) is currently supported via the ColumnPaginationFilter. However, it is not a very clean way of supporting pagination. Some of the problems with it are: * Normally, one would expect a query with (Filter(A) AND Filter(B)) to have same results as (query with Filter(A)) INTERSECT (query with Filter(B)). This is not the case for ColumnPaginationFilter as its internal state gets updated depending on whether or not Filter(A) returns TRUE/FALSE for a particular cell. * When this Filter is used in combination with other filters (e.g., doing AND with another filter using FilterList), the behavior of the query depends on the order of filters in the FilterList. This is not ideal. * ColumnPaginationFilter is a stateful filter which ends up counting multiple versions of the cell as separate values even if another filter upstream or the ScanQueryMatcher is going to reject the value for other reasons. Seems like we need a reliable way to do pagination. The particular use case that prompted this JIRA is pagination within the same rowKey. For example, for a given row key R, get columns with prefix P, starting at offset X (among columns which have prefix P) and limit Y. Some possible fixes might be: 1) enhance ColumnPrefixFilter to support another constructor which supports limit/offset. 2) Support pagination (limit/offset) at the Scan/Get API level (rather than as a filter) [Like SQL]. Original Post: Thanks Jiakai Liu for reporting this issue and doing the initial investigation. Email from Jiakai below: Assuming that we have an index column family with the following entries: tag0:001:thread1 ... tag1:001:thread1 tag1:002:thread2 ... tag1:010:thread10 ... tag2:001:thread1 tag2:005:thread5 ... To get threads with tag1 in range [5, 10), I tried the following code: ColumnPrefixFilter filter1 = new ColumnPrefixFilter(Bytes.toBytes(tag1)); ColumnPaginationFilter filter2 = new ColumnPaginationFilter(5 /* limit */, 5 /* offset */); FilterList filters = new FilterList(Operator.MUST_PASS_ALL); filters.addFilter(filter1); filters.addFilter(filter2); Get get = new Get(USER); get.addFamily(COLUMN_FAMILY); get.setMaxVersions(1); get.setFilter(filters); Somehow it didn't work as expected. It returned the entries as if the filter1 were not set. Turns out the ColumnPrefixFilter returns SEEK_NEXT_USING_HINT in some cases. The FilterList filter does not handle this return code properly (treat it as INCLUDE). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252966#comment-13252966 ] Phabricator commented on HBASE-5776: tedyu has commented on the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' - 'and there is no successful put' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared - Prepare src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' - caught src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk. src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount - successfulCount src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199 src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping. Should we introduce some counter for the above case? src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please. REVISION DETAIL https://reviews.facebook.net/D2775 HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252965#comment-13252965 ] Phabricator commented on HBASE-5776: tedyu has commented on the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' - 'and there is no successful put' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared - Prepare src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' - caught src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk. src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount - successfulCount src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199 src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping. Should we introduce some counter for the above case? src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please. REVISION DETAIL https://reviews.facebook.net/D2775 HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253056#comment-13253056 ] Phabricator commented on HBASE-5684: mbautin has commented on the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 This is actually a stability fix. We have seen the load balancer shedding regions from most loaded regionservers in some tests. REVISION DETAIL https://reviews.facebook.net/D2757 BRANCH make_processbasedlocalhbasecluster_run_HBASE-5684_v11 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251860#comment-13251860 ] Phabricator commented on HBASE-5684: Liyin has commented on the revision [jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. Mikhail, nice work:) LGTM ! REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_HBASE-5684_v10 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251867#comment-13251867 ] Phabricator commented on HBASE-5684: amirshim has commented on the revision [jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. Looks good. As we talked, maybe we should use [[http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html | WatchService]] for log tailer once we decide to move to Java 7. And we might want to split HBaseTestingUtility.java at some point since it's over 1500 lines. REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_HBASE-5684_v10 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251897#comment-13251897 ] Phabricator commented on HBASE-5763: stack has commented on the revision [jira] [HBASE-5763] [89-fb] Fix random failures in TestFSErrorsExposed. lgtm REVISION DETAIL https://reviews.facebook.net/D2739 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch, D2739.3.patch, D2739.4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252098#comment-13252098 ] Phabricator commented on HBASE-5763: amirshim has accepted the revision [jira] [HBASE-5763] [89-fb] Fix random failures in TestFSErrorsExposed. Looks good REVISION DETAIL https://reviews.facebook.net/D2739 BRANCH fix_random_failures_in_HBASE-5763_v5 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch, D2739.3.patch, D2739.4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252152#comment-13252152 ] Phabricator commented on HBASE-5684: tedyu has accepted the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 Is this restriction needed ? REVISION DETAIL https://reviews.facebook.net/D2757 BRANCH make_processbasedlocalhbasecluster_run_HBASE-5684_v11 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch, D2709.4.patch, D2757.1.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250888#comment-13250888 ] Phabricator commented on HBASE-5684: mbautin has commented on the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. Addressing Ted's comments. I will post a new version shortly. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Even though ports for same-type servers in the process-based cluster are selected randomly, it would still make sense to call the lowest-port server #0, the next-lowest #1, etc. src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 Done. src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Done. src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Done. REVISION DETAIL https://reviews.facebook.net/D2709 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250952#comment-13250952 ] Phabricator commented on HBASE-5684: tedyu has commented on the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. Patch looks good. I was warned that there was unit test problem when I tried to accept. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java:24 This test doesn't have assertion. Is it possible to merge it into other test classes ? REVISION DETAIL https://reviews.facebook.net/D2709 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251157#comment-13251157 ] Phabricator commented on HBASE-5684: tedyu has accepted the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. I still see 'This diff has Unit Test Problems.' REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_HBASE-5684_v9 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251160#comment-13251160 ] Phabricator commented on HBASE-5684: mbautin has commented on the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. Ted: do you mean Unit Tests Skipped at the top of the diff? We currently don't have integration between HBase unit tests and Arcanist/Phabricator. REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_HBASE-5684_v9 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch, D2709.2.patch, D2709.3.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5763) Fix random failures in TestFSErrorsExposed
[ https://issues.apache.org/jira/browse/HBASE-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251350#comment-13251350 ] Phabricator commented on HBASE-5763: stack has commented on the revision [jira] [HBASE-5763] [89-fb] Fix random failures in TestFSErrorsExposed. Patch looks good but for the comment below. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java:204 Do we have to wait ten seconds like this? Can we not poll somewhere to figure them up? REVISION DETAIL https://reviews.facebook.net/D2739 Fix random failures in TestFSErrorsExposed -- Key: HBASE-5763 URL: https://issues.apache.org/jira/browse/HBASE-5763 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2739.1.patch, D2739.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250007#comment-13250007 ] Phabricator commented on HBASE-5744: mbautin has abandoned the revision [jira] [HBASE-5744] Thrift server metrics should be long instead of int. Unfortunately, I committed this diff under the 89-fb differential revision first (D2679) and had to edit the commit message. Here is what I did: svn propedit -r 1311167 --revprop svn:log The new commit message is as follows: svn log -r1311167 https://svn.apache.org/repos/asf/hbase/trunk r1311167 | mbautin | 2012-04-09 02:00:11 -0700 (Mon, 09 Apr 2012) | 23 lines [jira] [HBASE-5744] Thrift server metrics should be long instead of int Summary: As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. There is a bug where we truncate a nanosecond latency to int, which is a problem with RPCs that take more than 2.147483647 seconds to process. This is a trunk diff. The 89-fb version of this diff is at D2679. Test Plan: TestThriftServer is updated to test for the failure case (an RPC is artificially made to take 3 seconds). The new test case fails without the fix. Re-run all unit tests. Reviewers: stack, sc, Kannan, JIRA Reviewed By: sc Differential Revision: https://reviews.facebook.net/D2685 If you have a git-svn checkout that got confused by this, here is how to get it back to a consistent state: git svn reset 1311166 git svn rebase Sorry about the inconvenience. REVISION DETAIL https://reviews.facebook.net/D2685 Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, D2685.3.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5684) Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust
[ https://issues.apache.org/jira/browse/HBASE-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250415#comment-13250415 ] Phabricator commented on HBASE-5684: tedyu has commented on the revision [jira] [HBASE-5684] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust. Patch looks good. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Should read 'to look up' src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Why is sorting needed ? src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 I think removing 'to kill' would make the sentence clearer. src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Should we add mapping for 'fs.defaultFS' as well ? REVISION DETAIL https://reviews.facebook.net/D2709 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust --- Key: HBASE-5684 URL: https://issues.apache.org/jira/browse/HBASE-5684 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D2709.1.patch Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249521#comment-13249521 ] Phabricator commented on HBASE-5744: mbautin has committed the revision [jira] [HBASE-5744] [89-fb] Thrift server metrics should be long instead of int. REVISION DETAIL https://reviews.facebook.net/D2679 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1310973 Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5730) [89-fb] Make HRegionThriftServer's thread pool bounded
[ https://issues.apache.org/jira/browse/HBASE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248619#comment-13248619 ] Phabricator commented on HBASE-5730: alevchuk has commented on the revision [jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded. When the thread count exceeds the limit, is the a counter that gets bumped every time a new thread is prevented from spawning? REVISION DETAIL https://reviews.facebook.net/D2643 BRANCH make_hregionthriftserver_s_thread_pool_HBASE-5730 [89-fb] Make HRegionThriftServer's thread pool bounded -- Key: HBASE-5730 URL: https://issues.apache.org/jira/browse/HBASE-5730 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer in HRegionThriftServer. We will address whatever problems HRegionThriftServer still has in trunk in HBASE-5703. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248936#comment-13248936 ] Phabricator commented on HBASE-5708: mbautin has committed the revision [jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests. REVISION DETAIL https://reviews.facebook.net/D2601 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1310612 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test -- Key: HBASE-5708 URL: https://issues.apache.org/jira/browse/HBASE-5708 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch, D2601.4.patch Some map-reduce-based tests are failing when executed concurrently in 89-fb because mini-map-reduce cluster uses /tmp/hadoop-username for temporary data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5730) [89-fb] Make HRegionThriftServer's thread pool bounded
[ https://issues.apache.org/jira/browse/HBASE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248937#comment-13248937 ] Phabricator commented on HBASE-5730: mbautin has committed the revision [jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded. REVISION DETAIL https://reviews.facebook.net/D2643 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1310613 [89-fb] Make HRegionThriftServer's thread pool bounded -- Key: HBASE-5730 URL: https://issues.apache.org/jira/browse/HBASE-5730 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer in HRegionThriftServer. We will address whatever problems HRegionThriftServer still has in trunk in HBASE-5703. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249144#comment-13249144 ] Phabricator commented on HBASE-5744: sc has accepted the revision [jira] [HBASE-5744] [89-fb] Thrift server metrics should be long instead of int. Looks good to me. Thanks a lot for the fix! REVISION DETAIL https://reviews.facebook.net/D2679 BRANCH thrift_server_metrics_should_be_long_HBASE-5744_v2 Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247095#comment-13247095 ] Phabricator commented on HBASE-5708: mbautin has commented on the revision [jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/OldLogsCleaner.java:98-100 Saw a NullPointerException on the next line. src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java:984-987 Saw an infinite loop with a SESSIONEXPIRED here in TestSplitLogManager. src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java:1070-1073 Saw an infinite loop with a SESSIONEXPIRED here in TestSplitLogManager. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:1591 Previously, we would proceed to process the open region message, even though we have re-queued it for later. src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java:52 A lot of stuff in this deprecated class should eventually be delegated to HBaseTestingUtility. src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:138-139 Sometimes we just want to reuse the mini-map-reduce-cluster-starting capabilities of HBaseTestingUtility, but provide a NameNode URI ourselves. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java:364 Observed failures to start namenode's trash cleaning thread when it failed to connect as a client to the newly started namenode. Adding a delay fixed the problem in some of those cases. src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java:80 The load balancer would unassign some regions in a load-test unit test, resulting in a failure, which is totally ridiculous. src/test/resources/hbase-site.xml:145 A lot of tests used to fail with ZK session expiration errors. I have set ZK session timeout to 1 sec in a few places where it seemed necessary. src/test/resources/hbase-site.xml:155 We did not have enough retries in some unit tests. This will hopefully add some robustness. REVISION DETAIL https://reviews.facebook.net/D2601 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test -- Key: HBASE-5708 URL: https://issues.apache.org/jira/browse/HBASE-5708 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch Some map-reduce-based tests are failing when executed concurrently in 89-fb because mini-map-reduce cluster uses /tmp/hadoop-username for temporary data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247328#comment-13247328 ] Phabricator commented on HBASE-5708: mbautin has commented on the revision [jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests. Here are some more results after running every test 10 times: - Before the patch: 76 test errors, 62 test failures, 17 tests without results - After the patch: 20 test errors, 22 test failures, 24 tests without results (2.34x or 57% reduction in total failures). The remaining tests will probably take some more thinking to fix. REVISION DETAIL https://reviews.facebook.net/D2601 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test -- Key: HBASE-5708 URL: https://issues.apache.org/jira/browse/HBASE-5708 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch Some map-reduce-based tests are failing when executed concurrently in 89-fb because mini-map-reduce cluster uses /tmp/hadoop-username for temporary data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5731) Make max line length 100 in linter
[ https://issues.apache.org/jira/browse/HBASE-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247483#comment-13247483 ] Phabricator commented on HBASE-5731: tedyu has accepted the revision [jira] [HBASE-5731] [89-fb] Make max line length 100 in linter. REVISION DETAIL https://reviews.facebook.net/D2625 BRANCH master Make max line length 100 in linter -- Key: HBASE-5731 URL: https://issues.apache.org/jira/browse/HBASE-5731 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2625.1.patch We have switched to 100 characters per line in our Java files. Making the change in the linter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5731) Make max line length 100 in linter
[ https://issues.apache.org/jira/browse/HBASE-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247493#comment-13247493 ] Phabricator commented on HBASE-5731: stack has commented on the revision [jira] [HBASE-5731] Make max line length 100 in linter. +1 REVISION DETAIL https://reviews.facebook.net/D2631 Make max line length 100 in linter -- Key: HBASE-5731 URL: https://issues.apache.org/jira/browse/HBASE-5731 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2625.1.patch, D2631.1.patch We have switched to 100 characters per line in our Java files. Making the change in the linter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5731) Make max line length 100 in linter
[ https://issues.apache.org/jira/browse/HBASE-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247505#comment-13247505 ] Phabricator commented on HBASE-5731: mbautin has committed the revision [jira] [HBASE-5731] [89-fb] Make max line length 100 in linter. REVISION DETAIL https://reviews.facebook.net/D2625 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH131 Make max line length 100 in linter -- Key: HBASE-5731 URL: https://issues.apache.org/jira/browse/HBASE-5731 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2625.1.patch, D2631.1.patch We have switched to 100 characters per line in our Java files. Making the change in the linter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5731) Make max line length 100 in linter
[ https://issues.apache.org/jira/browse/HBASE-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247504#comment-13247504 ] Phabricator commented on HBASE-5731: Kannan has accepted the revision [jira] [HBASE-5731] Make max line length 100 in linter. REVISION DETAIL https://reviews.facebook.net/D2631 BRANCH linter_100chars Make max line length 100 in linter -- Key: HBASE-5731 URL: https://issues.apache.org/jira/browse/HBASE-5731 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2625.1.patch, D2631.1.patch We have switched to 100 characters per line in our Java files. Making the change in the linter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5731) Make max line length 100 in linter
[ https://issues.apache.org/jira/browse/HBASE-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247519#comment-13247519 ] Phabricator commented on HBASE-5731: mbautin has committed the revision [jira] [HBASE-5731] Make max line length 100 in linter. REVISION DETAIL https://reviews.facebook.net/D2631 COMMIT https://reviews.facebook.net/rHBASE1310006 Make max line length 100 in linter -- Key: HBASE-5731 URL: https://issues.apache.org/jira/browse/HBASE-5731 Project: HBase Issue Type: New Feature Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2625.1.patch, D2631.1.patch We have switched to 100 characters per line in our Java files. Making the change in the linter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247619#comment-13247619 ] Phabricator commented on HBASE-5708: Liyin has accepted the revision [jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests. Looks good to me. Just one minor comments: INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/OldLogsCleaner.java:98-100 why not just return here? REVISION DETAIL https://reviews.facebook.net/D2601 BRANCH make_minimapredcluster_directory_a_HBASE-5708_v37 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test -- Key: HBASE-5708 URL: https://issues.apache.org/jira/browse/HBASE-5708 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch Some map-reduce-based tests are failing when executed concurrently in 89-fb because mini-map-reduce cluster uses /tmp/hadoop-username for temporary data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247636#comment-13247636 ] Phabricator commented on HBASE-5708: stack has commented on the revision [jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests. +1 I can take care of forward porting this great stuff Mikhail if you want. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java:52 Yes. These classes need to go away. src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java:72 This is one way of making the transition (smile). REVISION DETAIL https://reviews.facebook.net/D2601 BRANCH make_minimapredcluster_directory_a_HBASE-5708_v37 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test -- Key: HBASE-5708 URL: https://issues.apache.org/jira/browse/HBASE-5708 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch Some map-reduce-based tests are failing when executed concurrently in 89-fb because mini-map-reduce cluster uses /tmp/hadoop-username for temporary data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5730) [89-fb] Make HRegionThriftServer's thread pool bounded
[ https://issues.apache.org/jira/browse/HBASE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247844#comment-13247844 ] Phabricator commented on HBASE-5730: mbautin has commented on the revision [jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded. This also renames the bounded thread pool server for more consistency with trunk. REVISION DETAIL https://reviews.facebook.net/D2643 [89-fb] Make HRegionThriftServer's thread pool bounded -- Key: HBASE-5730 URL: https://issues.apache.org/jira/browse/HBASE-5730 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer in HRegionThriftServer. We will address whatever problems HRegionThriftServer still has in trunk in HBASE-5703. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247881#comment-13247881 ] Phabricator commented on HBASE-5708: Kannan has commented on the revision [jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:1585 excellent catch! REVISION DETAIL https://reviews.facebook.net/D2601 BRANCH make_minimapredcluster_directory_a_HBASE-5708_v38 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test -- Key: HBASE-5708 URL: https://issues.apache.org/jira/browse/HBASE-5708 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch, D2601.4.patch Some map-reduce-based tests are failing when executed concurrently in 89-fb because mini-map-reduce cluster uses /tmp/hadoop-username for temporary data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5730) [89-fb] Make HRegionThriftServer's thread pool bounded
[ https://issues.apache.org/jira/browse/HBASE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247904#comment-13247904 ] Phabricator commented on HBASE-5730: Kannan has commented on the revision [jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded. awesome... thanks for fixing this quickly Mikhail! REVISION DETAIL https://reviews.facebook.net/D2643 BRANCH make_hregionthriftserver_s_thread_pool_HBASE-5730 [89-fb] Make HRegionThriftServer's thread pool bounded -- Key: HBASE-5730 URL: https://issues.apache.org/jira/browse/HBASE-5730 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer in HRegionThriftServer. We will address whatever problems HRegionThriftServer still has in trunk in HBASE-5703. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5730) [89-fb] Make HRegionThriftServer's thread pool bounded
[ https://issues.apache.org/jira/browse/HBASE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247905#comment-13247905 ] Phabricator commented on HBASE-5730: Kannan has accepted the revision [jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java:283 I see: Hbase.Iface handler = new HBaseHandler(conf, metrics); handler = HbaseHandlerMetricsProxy.newInstance(handler, metrics, conf); in ThriftServer.java. Wondering if want to add this in. But it is ok if we handled metrics in the next pass too for the embedded thrift case. REVISION DETAIL https://reviews.facebook.net/D2643 BRANCH make_hregionthriftserver_s_thread_pool_HBASE-5730 [89-fb] Make HRegionThriftServer's thread pool bounded -- Key: HBASE-5730 URL: https://issues.apache.org/jira/browse/HBASE-5730 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer in HRegionThriftServer. We will address whatever problems HRegionThriftServer still has in trunk in HBASE-5703. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5730) [89-fb] Make HRegionThriftServer's thread pool bounded
[ https://issues.apache.org/jira/browse/HBASE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247915#comment-13247915 ] Phabricator commented on HBASE-5730: sc has commented on the revision [jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded. +1 looks good to me. REVISION DETAIL https://reviews.facebook.net/D2643 BRANCH make_hregionthriftserver_s_thread_pool_HBASE-5730 [89-fb] Make HRegionThriftServer's thread pool bounded -- Key: HBASE-5730 URL: https://issues.apache.org/jira/browse/HBASE-5730 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer in HRegionThriftServer. We will address whatever problems HRegionThriftServer still has in trunk in HBASE-5703. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5730) [89-fb] Make HRegionThriftServer's thread pool bounded
[ https://issues.apache.org/jira/browse/HBASE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247917#comment-13247917 ] Phabricator commented on HBASE-5730: mbautin has commented on the revision [jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java:283 Talked to Scott, and he says it is OK to omit the proxy wrapper. We will handle the metrics as the next step. REVISION DETAIL https://reviews.facebook.net/D2643 BRANCH make_hregionthriftserver_s_thread_pool_HBASE-5730 [89-fb] Make HRegionThriftServer's thread pool bounded -- Key: HBASE-5730 URL: https://issues.apache.org/jira/browse/HBASE-5730 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer in HRegionThriftServer. We will address whatever problems HRegionThriftServer still has in trunk in HBASE-5703. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247931#comment-13247931 ] Phabricator commented on HBASE-5708: Karthik has commented on the revision [jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:1591 Good catch indeed! The result would be that we would try to open the region, timeout on root and throw an exception which would cause it to get put on the todo queue again. So we would incur a latency. Wondering if something else would have been broken as a result? REVISION DETAIL https://reviews.facebook.net/D2601 BRANCH make_minimapredcluster_directory_a_HBASE-5708_v38 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test -- Key: HBASE-5708 URL: https://issues.apache.org/jira/browse/HBASE-5708 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch, D2601.4.patch Some map-reduce-based tests are failing when executed concurrently in 89-fb because mini-map-reduce cluster uses /tmp/hadoop-username for temporary data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5736) ThriftServerRunner.HbaseHandler.mutateRow() does not use ByteBuffer correctly
[ https://issues.apache.org/jira/browse/HBASE-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247984#comment-13247984 ] Phabricator commented on HBASE-5736: tedyu has accepted the revision HBASE-5736 [jira] ThriftServerRunner.HbaseHandler.mutateRow() does not use ByteBuffer correctly. If tests pass. REVISION DETAIL https://reviews.facebook.net/D2649 BRANCH get-bytes-thrift ThriftServerRunner.HbaseHandler.mutateRow() does not use ByteBuffer correctly - Key: HBASE-5736 URL: https://issues.apache.org/jira/browse/HBASE-5736 Project: HBase Issue Type: Bug Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5736.D2649.1.patch, HBASE-5736.D2649.2.patch We have fixed similar bug in https://issues.apache.org/jira/browse/HBASE-5507 It uses ByteBuffer.array() to read the ByteBuffer. This will ignore the offset return the whole underlying byte array. The bug can be triggered by using framed Transport thrift servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5700) [89-fb] Fix TestMiniClusterLoad* test failures
[ https://issues.apache.org/jira/browse/HBASE-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245540#comment-13245540 ] Phabricator commented on HBASE-5700: mbautin has committed the revision [jira] [HBASE-5700] [89-fb] Fix TestMiniClusterLoad* test failures. REVISION DETAIL https://reviews.facebook.net/D2583 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1309066 [89-fb] Fix TestMiniClusterLoad* test failures -- Key: HBASE-5700 URL: https://issues.apache.org/jira/browse/HBASE-5700 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2583.1.patch Porting TestMiniClusterLoad* tests to 89-fb in HBASE-5679 uncovered certain problems with mini-cluster setup in 89-fb that need to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5700) [89-fb] Fix TestMiniClusterLoad* test failures
[ https://issues.apache.org/jira/browse/HBASE-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244987#comment-13244987 ] Phabricator commented on HBASE-5700: Kannan has accepted the revision [jira] [HBASE-5700] [89-fb] Fix TestMiniClusterLoad* test failures. thanks! REVISION DETAIL https://reviews.facebook.net/D2583 BRANCH fix_testminiclusterload_test_failures_HBASE-5700 [89-fb] Fix TestMiniClusterLoad* test failures -- Key: HBASE-5700 URL: https://issues.apache.org/jira/browse/HBASE-5700 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2583.1.patch Porting TestMiniClusterLoad* tests to 89-fb in HBASE-5679 uncovered certain problems with mini-cluster setup in 89-fb that need to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[ https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236380#comment-13236380 ] Phabricator commented on HBASE-5591: stack has commented on the revision HBASE-5591 [jira] ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes(). INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:611 Funny. If it was added by you, then I'd say its your prerogative to remove it! Good stuff. REVISION DETAIL https://reviews.facebook.net/D2355 BRANCH getbytes ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes() - Key: HBASE-5591 URL: https://issues.apache.org/jira/browse/HBASE-5591 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Fix For: 0.96.0 Attachments: HBASE-5591.D2355.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[ https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236444#comment-13236444 ] Phabricator commented on HBASE-5591: sc has commented on the revision HBASE-5591 [jira] ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes(). INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:611 Sorry for the trouble. Thanks for bearing with me :) REVISION DETAIL https://reviews.facebook.net/D2355 BRANCH getbytes ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes() - Key: HBASE-5591 URL: https://issues.apache.org/jira/browse/HBASE-5591 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Fix For: 0.96.0 Attachments: HBASE-5591.D2355.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency to DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237193#comment-13237193 ] Phabricator commented on HBASE-5469: mbautin has committed the revision [jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool. REVISION DETAIL https://reviews.facebook.net/D2409 COMMIT https://reviews.facebook.net/rHBASE1304626 Add baseline compression efficiency to DataBlockEncodingTool Key: HBASE-5469 URL: https://issues.apache.org/jira/browse/HBASE-5469 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2409.1.patch, D2409.2.patch, jira-HBASE-5469-Add-baseline-compression-efficiency--2012-03-23_15_04_41.patch DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size). Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk) Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[ https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236007#comment-13236007 ] Phabricator commented on HBASE-5591: stack has commented on the revision HBASE-5591 [jira] ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes(). Otherwise patch looks good. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:611 Why do you think this comment is here Scott? Its ominous lookinh. REVISION DETAIL https://reviews.facebook.net/D2355 BRANCH getbytes ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes() - Key: HBASE-5591 URL: https://issues.apache.org/jira/browse/HBASE-5591 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Attachments: HBASE-5591.D2355.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency to DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236042#comment-13236042 ] Phabricator commented on HBASE-5469: mbautin has commented on the revision [jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool. Ted: thanks for the review! Everyone else: is this OK to commit? Could someone else +1 too? Thanks! REVISION DETAIL https://reviews.facebook.net/D2409 BRANCH add_baseline_compression_efficiency_to_HBASE-5469_v6 Add baseline compression efficiency to DataBlockEncodingTool Key: HBASE-5469 URL: https://issues.apache.org/jira/browse/HBASE-5469 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2409.1.patch, D2409.2.patch DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size). Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk) Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port
[ https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236134#comment-13236134 ] Phabricator commented on HBASE-5613: lhofhansl has commented on the revision HBASE-5613 [jira] ThriftServer getTableRegions does not return serverName and port. +1 REVISION DETAIL https://reviews.facebook.net/D2403 BRANCH 5613 ThriftServer getTableRegions does not return serverName and port Key: HBASE-5613 URL: https://issues.apache.org/jira/browse/HBASE-5613 Project: HBase Issue Type: Bug Components: thrift Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5613.D2403.1.patch, HBASE-5613.D2403.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port
[ https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236183#comment-13236183 ] Phabricator commented on HBASE-5613: sc has commented on the revision HBASE-5613 [jira] ThriftServer getTableRegions does not return serverName and port. Sorry. I just found that the original method supports the case that table doesn't exist (it returns an empty list). I am adding that logic here so it can be consistent. REVISION DETAIL https://reviews.facebook.net/D2403 BRANCH 5613 ThriftServer getTableRegions does not return serverName and port Key: HBASE-5613 URL: https://issues.apache.org/jira/browse/HBASE-5613 Project: HBase Issue Type: Bug Components: thrift Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port
[ https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236203#comment-13236203 ] Phabricator commented on HBASE-5613: lhofhansl has commented on the revision HBASE-5613 [jira] ThriftServer getTableRegions does not return serverName and port. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:589 Hmm... Now you're make two roundtrips. Does getTable throw an exception? Can we catch that instead? If not, not a big deal. REVISION DETAIL https://reviews.facebook.net/D2403 BRANCH 5613 ThriftServer getTableRegions does not return serverName and port Key: HBASE-5613 URL: https://issues.apache.org/jira/browse/HBASE-5613 Project: HBase Issue Type: Bug Components: thrift Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port
[ https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236226#comment-13236226 ] Phabricator commented on HBASE-5613: sc has commented on the revision HBASE-5613 [jira] ThriftServer getTableRegions does not return serverName and port. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:589 Good idea. We can catch TableNotExist and return empty list. I will make the change. Thanks! REVISION DETAIL https://reviews.facebook.net/D2403 BRANCH 5613 ThriftServer getTableRegions does not return serverName and port Key: HBASE-5613 URL: https://issues.apache.org/jira/browse/HBASE-5613 Project: HBase Issue Type: Bug Components: thrift Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
[ https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234478#comment-13234478 ] Phabricator commented on HBASE-5542: sc has commented on the revision HBASE-5542 [jira] Unify HRegion.mutateRowsWithLocks() and HRegion.processRow(). @Lars, @Ted: If HBASE-5542.2.txt looks good to you guys, can we commit it? Then we can move on to HBASE-5565? There is no point to sit here to long. The patch might be stale later. REVISION DETAIL https://reviews.facebook.net/D2217 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow() Key: HBASE-5542 URL: https://issues.apache.org/jira/browse/HBASE-5542 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.96.0 Attachments: HBASE-5542.2.txt, HBASE-5542.D2217.1.patch, HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt mutateRowsWithLocks() does atomic mutations on multiple rows. processRow() does atomic read-modify-writes on a single row. It will be useful to generalize both and have a processRowsWithLocks() that does atomic read-modify-writes on multiple rows. This also helps reduce some redundancy in the codes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
[ https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234490#comment-13234490 ] Phabricator commented on HBASE-5542: lhofhansl has accepted the revision HBASE-5542 [jira] Unify HRegion.mutateRowsWithLocks() and HRegion.processRow(). Yeah, let's commit this. I won't get to HBASE-5565 any time soon. Sorry for the hold up. REVISION DETAIL https://reviews.facebook.net/D2217 BRANCH 5542 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow() Key: HBASE-5542 URL: https://issues.apache.org/jira/browse/HBASE-5542 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.96.0 Attachments: HBASE-5542.2.txt, HBASE-5542.D2217.1.patch, HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt mutateRowsWithLocks() does atomic mutations on multiple rows. processRow() does atomic read-modify-writes on a single row. It will be useful to generalize both and have a processRowsWithLocks() that does atomic read-modify-writes on multiple rows. This also helps reduce some redundancy in the codes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
[ https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234601#comment-13234601 ] Phabricator commented on HBASE-5542: sc has commented on the revision HBASE-5542 [jira] Unify HRegion.mutateRowsWithLocks() and HRegion.processRow(). @Ted, @Lars: Thanks for reviewing this :) I appreciate it. REVISION DETAIL https://reviews.facebook.net/D2217 BRANCH 5542 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow() Key: HBASE-5542 URL: https://issues.apache.org/jira/browse/HBASE-5542 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.96.0 Attachments: HBASE-5542.2.txt, HBASE-5542.D2217.1.patch, HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt mutateRowsWithLocks() does atomic mutations on multiple rows. processRow() does atomic read-modify-writes on a single row. It will be useful to generalize both and have a processRowsWithLocks() that does atomic read-modify-writes on multiple rows. This also helps reduce some redundancy in the codes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port
[ https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235184#comment-13235184 ] Phabricator commented on HBASE-5613: tedyu has accepted the revision HBASE-5613 [jira] ThriftServer getTableRegions does not return serverName and port. If tests pass. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java:445 Is this for debugging ? REVISION DETAIL https://reviews.facebook.net/D2403 BRANCH 5613 ThriftServer getTableRegions does not return serverName and port Key: HBASE-5613 URL: https://issues.apache.org/jira/browse/HBASE-5613 Project: HBase Issue Type: Bug Components: thrift Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Attachments: HBASE-5613.D2403.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port
[ https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235192#comment-13235192 ] Phabricator commented on HBASE-5613: sc has commented on the revision HBASE-5613 [jira] ThriftServer getTableRegions does not return serverName and port. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java:445 Yes. But it's also good to print something during the test. REVISION DETAIL https://reviews.facebook.net/D2403 BRANCH 5613 ThriftServer getTableRegions does not return serverName and port Key: HBASE-5613 URL: https://issues.apache.org/jira/browse/HBASE-5613 Project: HBase Issue Type: Bug Components: thrift Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Attachments: HBASE-5613.D2403.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency to DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235241#comment-13235241 ] Phabricator commented on HBASE-5469: tedyu has accepted the revision [jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool. It would be nice to attach sample output from DataBlockEncodingTool to JIRA. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:53 Nice catch. src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:150 Nice. src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:166 This doesn't match the actual parameter name. src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:183 Where is cachedEncodedData invalidated ? REVISION DETAIL https://reviews.facebook.net/D2409 BRANCH add_baseline_compression_efficiency_to_HBASE-5469_v5 Add baseline compression efficiency to DataBlockEncodingTool Key: HBASE-5469 URL: https://issues.apache.org/jira/browse/HBASE-5469 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2409.1.patch DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size). Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk) Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235259#comment-13235259 ] Phabricator commented on HBASE-5335: mbautin has accepted the revision [jira] [HBASE-5335] Dynamic Schema Config. Nicolas: the most recent version looks good! I am assuming Ted, Stack, and others have also reviewed this in detail. REVISION DETAIL https://reviews.facebook.net/D2247 BRANCH 5335-89fb Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch, D2247.5.patch, D2247.6.patch, D2247.7.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233782#comment-13233782 ] Phabricator commented on HBASE-5335: mbautin has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. Nicolas: could you please re-submit this with lint results? You need to re-run mvn -Darc initialize, and then you can use arc diff --preview instead of arc diff --only to include lint results. REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch, D2247.5.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[
https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232697#comment-13232697
]
Phabricator commented on HBASE-5335:
nspiegelberg has commented on the revision [jira] [HBASE-5335] Dynamic Schema
Config.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:947 Important
notes:
1. CompoundConfiguration is a derived class from Configuration
2. there is a CompoundConfiguration.add(Configuration) function
HRegion.conf = CompoundConfiguration(BaseConf, HTD)
if you passed HRegion.conf to the daughter region constructor on a split, the
daughter region would have:
HRegion.conf
= CompoundConfiguration(CompoundConfiguration(BaseConf, HTD), HTD)
= CompoundConfiguration(BaseConf, HTD, HTD)
We need to dedupe the HTD.
src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:1 I
guess the question is: do we want to refactor the existing API both all conf
classes under a conf directory? I put this under util because I didn't want to
clutter the main folder with a class that was only supposed to be used
internally. I considered putting it under the regionserver folder. Maybe
that's a better fit?
src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java:169 yeah. another
option is making a copy constructor
src/main/ruby/hbase/admin.rb:187 Basically, your schema would look like:
NAME = 'blah', BLOOMFILTER = ROWCOL,
ADVANCED = {hbase.hstore.compaction.ratio = 0.25}
I don't want to explain the notion of ADVANCED too much beyond HBase
committers. Basically, it's only a toggle for people who know what they're
doing and aren't afraid to be power users and look at code. If we get really
common config patterns, we should pull them out to reserved keywords for common
users and then map. For example:
COMPACT_RATIO = 'hbase.hstore.compaction.ratio'
Why make a config option that most people won't play with? Because , as
power users, we can iterate on functionality help users. There is now a
workaround for a specific user's problem without modifying code and we don't
have to have advanced deprecation strategies like we would with a reserved
keyword.
REVISION DETAIL
https://reviews.facebook.net/D2247
Dynamic Schema Configurations
-
Key: HBASE-5335
URL: https://issues.apache.org/jira/browse/HBASE-5335
Project: HBase
Issue Type: New Feature
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Labels: configuration, schema
Attachments: D2247.1.patch, D2247.2.patch
Currently, the ability for a core developer to add per-table per-CF
configuration settings is very heavyweight. You need to add a reserved
keyword all the way up the stack you have to support this variable
long-term if you're going to expose it explicitly to the user. This has
ended up with using Configuration.get() a lot because it is lightweight and
you can tweak settings while you're trying to understand system behavior
[since there are many config params that may never need to be tuned]. We
need to add the ability to put read arbitrary KV settings in the HBase
schema. Combined with online schema change, this will allow us to safely
iterate on configuration settings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232824#comment-13232824 ] Phabricator commented on HBASE-5521: mbautin has accepted the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. REVISION DETAIL https://reviews.facebook.net/D2097 BRANCH svn Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Fix For: 0.96.0 Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.10.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, HBASE-5521.D2097.9.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232832#comment-13232832 ] Phabricator commented on HBASE-5521: mbautin has committed the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. REVISION DETAIL https://reviews.facebook.net/D2097 COMMIT https://reviews.facebook.net/rHBASE1302602 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Fix For: 0.96.0 Attachments: HBASE-5521-jira-Move-compression-decompression-to-an-2012-03-19_12_12_32.patch, HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.10.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, HBASE-5521.D2097.9.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232915#comment-13232915 ] Phabricator commented on HBASE-5335: mbautin has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. A few initial comments (unfortunately on an earlier version of the revision). INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:748 Use containsKey src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:753 Are we trying to make the output jruby-parseable? If so, we need to take care of escaping embedded single quotes here. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:760 HConstants.ADVANCED sounds confusing. I think we need that constant to have a more descriptive name. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:773-775 See my earlier comment about escaping embedded single quotes. The escaping method also needs to be shared between all callsites. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:783 Is it possible to create the unmodifiable map once instead of every time this is called? src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:41 I might have missed some earlier discussion, but why exactly is a Guava dependency a bad thing? Guava is licensed under Apache License 2.0 according to http://code.google.com/p/guava-libraries/. REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232946#comment-13232946 ] Phabricator commented on HBASE-5335: nspiegelberg has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:760 actually, the purpose was to dissuade people from using this unless they know what they're doing. we don't want people randomly putting keys in here without looking at the source code and then wondering why it doesn't work. make sense? do you have another suggestion? src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:753 we don't currently have any config variables with single quotes, do we? it's much more developer-controlled than user input. I guess I should do some basic sanitization for the user error case. the important ability is that the user can mistakenly enter a key with a single quote and then delete it src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:783 this is a shallow pointer copy, not a deep KV copy src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:41 @mbautin: ted's point is that this will be in client code, so every app server MapReduce cluster would need to have the Guava dependency installed versus just the HBase server deployment. REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232953#comment-13232953 ] Phabricator commented on HBASE-5335: mbautin has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. Nicolas: Looks good! Some more comments inline. Also, we now have lint enabled. Could you please re-run mvn -Darc initialize, then do arc lint or arc diff --preview and address lint comments? Thanks! INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/HConstants.java:361 This constant needs a better name. Probably some of the constants above it do, too. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java:530 Is it possible to reduce code duplication between here and HColumnDescriptor? src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java:549 single quote escaping src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java:579-581 single quote escaping src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java:367 Looks similar to the corresponding function in HColumnDescriptor. Is it possible to reuse code between the two? (getValues() would be a bigger case for that.) src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:42 Add p/ src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:222-229 Why exactly do we have to copy-and-paste the code instead of using composition or inheritance? If there is a bug in Configuration and it is fixed there, it will not be reflected here, which is indeed somewhat tragic. src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4246 It would be great to avoid adding more tests to TestFromClientSide and create a separate test class instead. src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4247 This probably should not use javadoc syntax. src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4312 Is it possible to wait for an event instead of a specific amount of time? src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:753 In that case we should probably sanitize user input for single quotes somewhere else, wherever the user is supposed to tweak configuration values in real time. However, I think it is easier to escape single quotes here. It would also be good to parse the output value of this function with jruby in a test. src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4322 Create an HConstant for this key src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4292 Sleep times should be according to http://hbase.apache.org/book.html#hbase.tests.writing . We should probably have a constant for each type of sleep time. Quoting the linked page from HBase book: Whenever possible, tests should not use Thread.sleep, but rather waiting for the real event they need. This is faster and clearer for the reader. Tests should not do a Thread.sleep without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performance is. Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. src/test/java/org/apache/hadoop/hbase/util/TestCompoundConfiguration.java:33 Can this be private? src/test/java/org/apache/hadoop/hbase/util/TestCompoundConfiguration.java:43-46 Is this override necessary? src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:760 I guess we need a javadoc saying that ADVANCED is not for general-purpose use, and that it is meant for use as an HColumnDescriptor key, then. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:783 OK, if you think we won't be calling this a lot, this is fine with me. REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232968#comment-13232968 ] Phabricator commented on HBASE-5335: nspiegelberg has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:753 I looked into this some more. We would need to escape more than just single quotes. There is a function StringEscapeUtils.escapeJavaScript() in apache.commons.lang that uses the same parsing escapes as Ruby. However, that requires us to either ensure client packaging of this dependency or write a custom parser for this. practically, single quotes are a mistake. we offer zero parsing sanitization as is and just let the RegionServer throw an IllegalArgumentException if the user configures it wrong. we need to let the user undo a single quote mistake. this is currently possible. value sanitization is the same as existing functionality src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:760 currently, there is no javadoc info the user has no way of knowing he can do this unless he knows about it from the JIRA. security through obscurity. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java:367 Already talked with Ted Stack about this in previous comments. A summary: Both HTD HCD have massive redundancies that could be unified beyond this function. They are both basically decorated K,V stores. I don't think unification is very necessary until we have something like locality groups and need a third K,V store. Have you traditionally seen bugs related to HTD HCD inconsistencies? I'm just worried that there is time spent on thinking of how to refactor this code without thinking about whether the denormalized design is actually hurting us on a practical feature design/debug sense. src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:222-229 3 problems: 1) If I don't use Configuration.java, then I need to refactor a LOT of the regionserver code to use the new API 2) we basically need to override Configuration.getProps or we'd need protected access to Configuration.properties so we can modify that pointer. Basically, there are a bunch of functions in the base Configuration that call getProps() and we need to use our derived version instead of the base version. 3) Trying to make a generic implementation that works across all HDFS versions. I would like to modify Configuration.properties in HDFS 1.0 to be protected, but I'd still need to have this code until my patch made it to all the versions we support. Configuration.java hasn't changed much, so I don't think this is an issue in practice. Do you have another strategy? src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4292 The 10 sec sleep is found from TestFromClientSide.compactCFRegion. I used the same paradigm here and added poll-waiting to speed up the test. We need the synchronous compaction feature before we can remove this (HBASE-2949). REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232978#comment-13232978 ] Phabricator commented on HBASE-5335: mbautin has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:222-229 OK, got it. Please note in the comment that these methods are private in some HDFS versions HBase has to support. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java:367 OK, sounds good. REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232980#comment-13232980 ] Phabricator commented on HBASE-5335: nspiegelberg has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4246 we need to chop this file into multiple pieces for parallelization, separate from this JIRA, correct? testFromClientSide seems to be the correct spot since it's an HBase java client integration test. src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4326 will make an HConstant REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232985#comment-13232985 ] Phabricator commented on HBASE-5335: nspiegelberg has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:222-229 will do. REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232990#comment-13232990 ] Phabricator commented on HBASE-5335: mbautin has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. A couple more responses inline. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4246 This would be the right place for a new test like this if it were not as huge as it is now. Would it be difficult to add new unit tests into separate test classes? I have tried to do so when creating new tests and have not had much difficulties. src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:4292 It would be great to add this explanation to the comment and/or make this delay a meaningfully-named constant and reuse it in both places. REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232334#comment-13232334 ] Phabricator commented on HBASE-5521: heyongqiang has commented on the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. i think Mikhail already did +1 on this. The last few updates are just to address tedyu's comments REVISION DETAIL https://reviews.facebook.net/D2097 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Fix For: 0.96.0 Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.10.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, HBASE-5521.D2097.9.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231890#comment-13231890 ] Phabricator commented on HBASE-5521: heyongqiang has commented on the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:61 The interface is there before this change. I agree it need a better name. But i think that should be done in a different context. src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java:109 this interface is removed from the interface as to address your previous comments. Methods defined in an interface need to be clear and need to be as fewer as possible. Once it is added, it is not easy to take back. src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockEncodingContext.java:34 Yes. It returns encoded uncompressed bytes. src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:202 ok. i will rename it to encodeData() src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java:106 will fix src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:56 nope. it's 'as well as'...will put a few more words here REVISION DETAIL https://reviews.facebook.net/D2097 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231984#comment-13231984 ] Phabricator commented on HBASE-5521: tedyu has commented on the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. Looks good. I have only one comment outstanding w.r.t. compressKeyValues() INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:61 I looked at PrefixKeyDeltaEncoder.compressKeyValues(DataOutputStream ...) It only does encoding (and appending memstoreTs). I think we should take this opportunity to make interface method name(s) clearer. src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java:109 Pardon me if my comments have been inconsistent. REVISION DETAIL https://reviews.facebook.net/D2097 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, HBASE-5521.D2097.9.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231987#comment-13231987 ] Phabricator commented on HBASE-5521: tedyu has commented on the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockEncodingContext.java:34 Can we make this javadoc clearer ? Something like: @return encoded bytes which are ready ... This would leave no confusion as to when encoding is done. src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockEncodingContext.java:39 Similar to comment above. When 'after encoding' is placed at the end of the sentence, I am not sure interpretation of where encoding is done would be uniform. REVISION DETAIL https://reviews.facebook.net/D2097 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Fix For: 0.96.0 Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, HBASE-5521.D2097.9.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232101#comment-13232101 ] Phabricator commented on HBASE-5521: heyongqiang has commented on the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. will address the java doc etc comments. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:61 It will also do the compression. The internalEncodeKeyValues() only does the encoding. but compressKeyValues will do encoding (by calling internalEncode...) and compress in postEncoding(). REVISION DETAIL https://reviews.facebook.net/D2097 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Fix For: 0.96.0 Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, HBASE-5521.D2097.9.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232149#comment-13232149 ] Phabricator commented on HBASE-5521: tedyu has commented on the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. Looks good. I will defer the acceptance decision to Mikhail. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockEncodingContext.java:40 Should we say 'uncompressed' instead of 'not heavily compressed' here so that the javadoc is consistent with the name of the method ? REVISION DETAIL https://reviews.facebook.net/D2097 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Fix For: 0.96.0 Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.10.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, HBASE-5521.D2097.9.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5575) Configure Arcanist lint engine for HBase
[ https://issues.apache.org/jira/browse/HBASE-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231613#comment-13231613 ] Phabricator commented on HBASE-5575: mbautin has committed the revision [jira] [HBASE-5575] Configure Arcanist lint engine for HBase. REVISION DETAIL https://reviews.facebook.net/D2289 COMMIT https://reviews.facebook.net/rHBASE1301751 Configure Arcanist lint engine for HBase Key: HBASE-5575 URL: https://issues.apache.org/jira/browse/HBASE-5575 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: Enabling-lint-2012-03-16_13_40_37.patch We need to enable Arcanist lint engine in HBase, so that a commit could be checked by running arc lint. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231644#comment-13231644 ] Phabricator commented on HBASE-5521: mbautin has commented on the revision HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context. Yongqiang: we now have a linter available in HBase trunk. Could you please run arc lint, resolve lint warnings, and resubmit the diff with arc diff --preview? REVISION DETAIL https://reviews.facebook.net/D2097 Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231671#comment-13231671 ] Phabricator commented on HBASE-5335: tedyu has commented on the revision [jira] [HBASE-5335] Dynamic Schema Config. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:65 This seems to be the only place com.google.common.collect.Lists is used. Can we instantiate ArrayList directly ? src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:120 This can be brought onto line 119. src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:143 Should getRaw() be called here ? src/main/java/org/apache/hadoop/hbase/util/CompoundConfiguration.java:153 This can be removed. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java:526 Can this be unified with HColumnDescriptor.getValues() ? REVISION DETAIL https://reviews.facebook.net/D2247 Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Attachments: D2247.1.patch, D2247.2.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5581) Creating a table with invalid syntax does not give an error message when it fails
[
https://issues.apache.org/jira/browse/HBASE-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231678#comment-13231678
]
Phabricator commented on HBASE-5581:
mbautin has committed the revision [jira] [HBASE-5581] [89-fb] Creating a
table with invalid syntax does not give an error message when it fails.
REVISION DETAIL
https://reviews.facebook.net/D2343
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1301789
Creating a table with invalid syntax does not give an error message when it
fails
-
Key: HBASE-5581
URL: https://issues.apache.org/jira/browse/HBASE-5581
Project: HBase
Issue Type: Bug
Components: shell
Reporter: Binu John
Priority: Minor
Fix For: 0.94.0, 0.96.0
Attachments: 5581trunk.patch, D2343.1.patch
Creating a table with invalid syntax does not give an error message when it
fails. In this case, it doesn't actually create the CF requested, but doesn't
give any indication to the user that it failed.
create 'test', {NAME = 'test', VERSIONS = 1, BLOCKCACHE = true, NUMREGIONS
= 20, SPLITALGO = HexStringSplit, COMPRESSION = 'LZO', BLOOMFILTER =
'ROW'}
0 row(s) in 3.0930 seconds
hbase(main):002:0 describe 'test'
DESCRIPTION
ENABLED
{NAME = 'test', FAMILIES = []}
true
1 row(s) in 0.1430 seconds
Putting {NUMREGIONS = 20, SPLITALGO = HexStringSplit} into a separate
stanza works fine, so the feature is fine.
create 'test', {NAME = 'test', VERSIONS = 1, BLOCKCACHE = true,
COMPRESSION = 'LZO', BLOOMFILTER = 'ROW'}, {NUMREGIONS = 20, SPLITALGO =
HexStringSplit}
0 row(s) in 2.7860 seconds
hbase(main):002:0 describe 'test'
DESCRIPTION
ENABLED
{NAME = 'test', FAMILIES = [{NAME = 'test', DATA_BLOCK_ENCODING =
'NONE', true
BLOOMFILTER = 'ROW', REPLICATION_SCOPE = '0', COMPRESSION = 'LZO',
VERSIONS
= '1', TTL = '2147483647', BLOCKSIZE = '65536', BLOOMFILTER_ERRORRATE =
'
0.01', ENCODE_ON_DISK = 'true', IN_MEMORY = 'false', BLOCKCACHE =
'true'}]}
We should throw an error if we can't create the CF so it's clear to the user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
