[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495318#comment-13495318
]
nkeywal commented on HBASE-5937:
In FSHLog, a wrong reference to getReader survived in the javadoc:
{noformat}
* To read an HLog, call {@link #getReader(org.apache.hadoop.fs.FileSystem,
* org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration)}.
{noformat}
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Fix For: 0.96.0
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
HBASE-5937.v2.patch, HBASE-5937.v3.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495323#comment-13495323 ] Flavio Junqueira commented on HBASE-5937: - [~nkeywal] Please create a subtask under HBASE-6931. Thanks for reporting! Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Fix For: 0.96.0 Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, HBASE-5937.v3.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495334#comment-13495334 ] nkeywal commented on HBASE-5937: Hi Flavio, Done, it's in HBASE-6931 / HBASE-7146 Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Fix For: 0.96.0 Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, HBASE-5937.v3.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467575#comment-13467575
]
Hadoop QA commented on HBASE-5937:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547291/HBASE-5937.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 72 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
157 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 6 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHCM
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2984//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2984//console
This message is automatically generated.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
HBASE-5937.v2.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467589#comment-13467589 ] Flavio Junqueira commented on HBASE-5937: - I'll add an issue to the umbrella jira to revisit javadocs. I don't see how TestHCM could possibly have failed because of this patch. I'll fix the findbugs warnings. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467813#comment-13467813
]
Hadoop QA commented on HBASE-5937:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547389/HBASE-5937.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 72 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
157 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2985//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2985//console
This message is automatically generated.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
HBASE-5937.v2.patch, HBASE-5937.v3.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467829#comment-13467829
]
Ted Yu commented on HBASE-5937:
---
Nice to see a green QA run.
{code}
-@InterfaceAudience.Private
-public class HLog implements Syncable {
- static final Log LOG = LogFactory.getLog(HLog.class);
- public static final byte [] METAFAMILY = Bytes.toBytes(METAFAMILY);
- static final byte [] METAROW = Bytes.toBytes(METAROW);
+public interface HLog {
{code}
Please restore annotation for HLog. I suggest using Public + Evolving.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
HBASE-5937.v2.patch, HBASE-5937.v3.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467855#comment-13467855 ] stack commented on HBASE-5937: -- I can restore the annotation on commit. Will leave it private for now. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, HBASE-5937.v3.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468066#comment-13468066
]
Ted Yu commented on HBASE-5937:
---
I am confused as to the scope of the commit:
{code}
$ svn log
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
| head
r1393126 | stack | 2012-10-02 12:29:19 -0700 (Tue, 02 Oct 2012) | 1 line
HBASE-5699 Refactor HLog into an interface
{code}
First, JIRA number was wrong.
Second, LongTermArchivingHFileCleaner.java wasn't even in Flavio's patch.
Maybe one version of patch for HBASE-6707 was integrated by chance ?
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Fix For: 0.96.0
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
HBASE-5937.v2.patch, HBASE-5937.v3.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468134#comment-13468134 ] stack commented on HBASE-5937: -- Updated the svn commit log to point to this issue instead of parent issue. Also undid the hbase-6707 pollution accidentally committed. Thanks for fingering this Ted. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Fix For: 0.96.0 Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, HBASE-5937.v3.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468178#comment-13468178 ] Hudson commented on HBASE-5937: --- Integrated in HBase-TRUNK #3409 (See [https://builds.apache.org/job/HBase-TRUNK/3409/]) HBASE-5937 Refactor HLog into an interface; REMOVE HBASE-6707 POLLUTION/OVERCOMMIT (Revision 1393221) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Fix For: 0.96.0 Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, HBASE-5937.v3.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468184#comment-13468184 ] Hudson commented on HBASE-5937: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #204 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/204/]) HBASE-5937 Refactor HLog into an interface; REMOVE HBASE-6707 POLLUTION/OVERCOMMIT (Revision 1393221) HBASE-5937 Refactor HLog into an interface (Revision 1393126) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/Compressor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogMetrics.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java *
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466886#comment-13466886
]
Flavio Junqueira commented on HBASE-5937:
-
Thanks a lot for the review, Stack, and Ted for the additions. Here are some
more comments:
bq. How is HLogSplitter going to work w/ different implementations? Currently
you punt and refer to FSHLog?
The splitter might not be necessary for every implementation, so I don't think
it is a good idea to make it part of HLog. Since BookKeeper is optimized for
writing multiple parallel streams efficiently, my perspective is that we want
to benefit from this feature to avoid having to split upon recovery. This is my
understanding at least, and it would be great to have a clarification if it is
incorrect.
If we assume that a splitter is necessary for every filesystem-based wal, then
we could have another interface extending HLog and defining methods related to
splitting. If not, then the splitter could specific to FSHLog. How does it
sound?
bq. Yeah, whats this crazy thing supposed to do? resetLogReaderClass
I can't remember now where I got it from, but it is used in TestHLogSplit only
to reset the class we use for the HLog reader. We have the ability to configure
the class we use for the reader and the writer.
bq. FYI, next time, you could use Bytes.toBytes in HBase under util to do this:
HBASE::CACHEFLUSH.getBytes(HConstants.UTF8_ENCODING);
Ok, thanks for the hint. I don't think I have added that statement, but I'm
happy to change it.
{quote}
The following from HLogUtil I'd think belong in an implementation:
validateHLogFilename
getReader // Why this in util and in factory too? or, its commented out
along w/ createWriter?
getHLogDirectoryName
getRegionDirRecoveredEditsDir // Can do this later... seems like I ask the
implementation for reader on the edits...
moveAsideBadEditsFile
getSplitEditFilesSorted // Yeah, this would be in the HLog splitting
implementation... or not exposed
{quote}
Are you suggesting that we make them static methods of say FSHLog instead of
having them in HLogUtil? They seem to be specific to the current trunk
implementation of HLog, so it seems right to do it. The idea behind HLogUtil
was to have all static methods in HLog originally. I can either move them now
or create a jira to revisit HLogUtil.
{quote}
So, what you want to do? If tests pass and you are fairly sure you have not
done anything to change our wal write speed – it doesn't look like you have
changed any of that, then if you want to do a part 1 commit and then open new
issue to address outstanding, improving WAL Interface, etc., I'd be +1 on that.
{quote}
I don't think we have changed anything that would affect your wal speed, at
least not consciously. The follow-up task I believe we need to work on, and
they could be jiras directly, are:
# Rename HLog interface to WAL as you suggested;
# Revisit methods of the HLog interface;
# Revisit methods of HLogUtil and HLogMetrics;
# Revisit HLogSplitter.
# Implement BKHLog.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466953#comment-13466953 ] stack commented on HBASE-5937: -- bq. The splitter might not be necessary for every implementation, so I don't think it is a good idea to make it part of HLog. Its fine it doesn't need to split but Master needs some way of figuring that w/ a particular implemenation, there is no need to split. How will you do this? bq. I can't remember now where I got it from, but it is used in TestHLogSplit only to reset the class we use for the HLog reader. We have the ability to configure the class we use for the reader and the writer. If only for that test, I'd say we file an issue to fix this ugly addition to HLog only used in a single test bq. Ok, thanks for the hint. I don't think I have added that statement, but I'm happy to change it. Not important. Just an FYI. bq. Are you suggesting that we make them static methods of say FSHLog instead of having them in HLogUtil? They seem to be me like they belong in the Interface... getReader... or, if FSHLog particular, in FSHLog. bq. The idea behind HLogUtil was to have all static methods in HLog originally. To my mind, class named HLogUtil is utility that could be used by any HLog implementation. If that is not the case, call it FSHLogUtil? You want to do your list in other JIRAs and get this patch committed now (do you want to do one last update)? Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466998#comment-13466998 ] Enis Soztutar commented on HBASE-5937: -- bq. I think I said this in another jira, but the plan we proposed was to make of HLog an interface and expose all public methods first. This way it will be simpler to revisit the current public methods and redesign the interface. Making of HLog an interface and redesigning it all at once might be too messy. I was also gonna suggest to abstract away implementation specific method calls (splits, etc), but agreed that this patch may become unmanageable. Let's do it your way of just extracting the interface, then we can work on the public api of the hlog. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467232#comment-13467232 ] Flavio Junqueira commented on HBASE-5937: - bq. Its fine it doesn't need to split but Master needs some way of figuring that w/ a particular implemenation, there is no need to split. How will you do this? I was wondering if the master really needs to be aware of splitting. To me it sounds like a feature that is implementation-specific. The master should be able to request the log stream for a region independent of how it gets it, no? Am I missing anything? bq. If only for that test, I'd say we file an issue to fix this ugly addition to HLog only used in a single test Agreed, it is not pretty. I can create a jira to get rid of it. bq. To my mind, class named HLogUtil is utility that could be used by any HLog implementation. If that is not the case, call it FSHLogUtil? Right now it makes no difference calling it HLogUtil or FSHLogUtil because even the interface HLog is heavily file-oriented. I wouldn't discard calling it FSHLogUtil and creating an HLogUtil in the case there are methods that are implementation-independent. Right now they all seem to be tied to FSHLog somehow. I also see no issue currently with moving those methods to FSHLog directly. Originally, having HLog, HLogUtil, FSHLog was just to have an interface, an implementation of the interface, and the static methods related for implementations of the interface. Separating them seems to make clear that the static methods are FSHLog specific. I'd like to have another careful look just to make sure I'm not missing anything. It could be another jira too to decide if we call it FSHLogUtil or merge those methods with FSHLog. bq. You want to do your list in other JIRAs and get this patch committed now (do you want to do one last update)? Sounds like a good idea because we can then focus on more concrete issue to refine it. It will make it more manageable. I'll do one more update shortly. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467249#comment-13467249 ] stack commented on HBASE-5937: -- bq. To me it sounds like a feature that is implementation-specific. The master should be able to request the log stream for a region independent of how it gets it, no? Ideally it would work this way but the fs log splitting is a heavy-duty process that farms logs out to the cluster for it to split. Might be hard 'hiding' this big old process behind a request for edits. Ok. Waiting on next update. Will commit only if you promise to do the cleanup (smile) else we'll be in this half-way state that will be hard to explain to the poor folks who come after us. Good on you Flavio. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467253#comment-13467253 ] Flavio Junqueira commented on HBASE-5937: - Sure, I can work on the cleanup. Hopefully it won't take long to converge to a state that is good for everyone. Let me know if you want me to create a new umbrella jira for all pending issues or if you have some other preference. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467295#comment-13467295 ] Andrew Purtell commented on HBASE-5937: --- +1 on approach and using an umbrella to group followups. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.v2.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466499#comment-13466499 ] Flavio Junqueira commented on HBASE-5937: - [~yuzhih...@gmail.com] bq. Looking at HLog interface, I wonder if interface Reader and interface Writer should be moved to their own files. That way we don't need to reference org.apache.hadoop.fs.FileSystem in HLog. I was wondering the same thing. One reason for not doing it was having all interfaces to implement for HLog in the same place. But, I don't feel very strongly about leaving them in HLog. [~stack] bq. You fellas like how the new Interface looks or would you change it? Stuff like isLowReplicationRollEnabled, getCoprocessorHost... these seems implementation specific. I think I said this in another jira, but the plan we proposed was to make of HLog an interface and expose all public methods first. This way it will be simpler to revisit the current public methods and redesign the interface. Making of HLog an interface and redesigning it all at once might be too messy. The bottom line is that we strongly agree that we need to revisit the methods currently in the interface and we need to do it eventually. bq. You have to do this cast? HBaseTestingUtility.setMaxRecoveryErrorCount(((FSHLog) wal2).getOutputStream(), 1); Well spotted. getOutputStream is currently not part of the interface because it is not a public method of FSHLog. According to the javadoc, it is used internally in FSHLog and in tests. Given that this is used in tests, we possibly don't want to make it specific to a particular implementation, although the test currently can't have anything else other than FSHLog for an instance of HLog. At the same time, getOutputStream returns a java.io.OutputStream, which is not specific to HDFS. I could really go both ways at this point. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466591#comment-13466591
]
stack commented on HBASE-5937:
--
How is HLogSplitter going to work w/ different implementations? Currently you
punt and refer to FSHLog?
This too seems like an HLog thing rather than something that belongs out in
util:
{code}
HLogUtil.getRegionDirRecoveredEditsDir
{code}
When region server opens, it'll want to ask the HLog implementation how to read
back its edits? Not go to util?
FYI in hbase, like hadoop, we're about two spaces for indent (See HLogMetrics)
Yeah, whats this crazy thing supposed to do? {code} resetLogReaderClass{code}
Factory looks like it needs a get reader for my edits and you pass in the
region whose edits you want?
FYI, next time, you could use Bytes.toBytes in HBase under util to do this:
HBASE::CACHEFLUSH.getBytes(HConstants.UTF8_ENCODING);
The following from HLogUtil I'd think belong in an implementation:
{code}
validateHLogFilename
getReader // Why this in util and in factory too? or, its commented out
along w/ createWriter?
getHLogDirectoryName
getRegionDirRecoveredEditsDir // Can do this later... seems like I ask the
implementation for reader on the edits...
moveAsideBadEditsFile
getSplitEditFilesSorted // Yeah, this would be in the HLog splitting
implementation... or not exposed at all
{code}
No problem if you want to address the above later
Something is up w/ your pom-it in the patch. You are removing its contents.
So, what you want to do? If tests pass and you are fairly sure you have not
done anything to change our wal write speed -- it doesn't look like you have
changed any of that, then if you want to do a part 1 commit and then open new
issue to address outstanding, improving WAL Interface, etc., I'd be +1 on that.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466165#comment-13466165 ] Flavio Junqueira commented on HBASE-5937: - Thanks for checking it out, Ted. I checked the findbugs warnings, and I couldn't find anything related to this patch, so the new patch I uploaded does not fix any findbugs warning. bq. Year is not needed. Sure, I have removed the three instances I found. bq. Please add class javadoc and annotation for audience. For this patch, I have simply copied the signatures of public methods to the HLog interface. If there is no javadoc, it means that there was no javadoc for the method implementation there before the patch. But, if you'd like us to create them for this patch, I don't have a problem working on it. To be clear, I'm just pointing out that it was not there before, and in general I agree they should have javadocs. bq. review board would help reviewers a lot. Agreed, I tried to create a review request, but I'm getting an internal server error right now, so I'll try later again. Hopefully it is a transient problem. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466166#comment-13466166 ] Hadoop QA commented on HBASE-5937: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547110/HBASE-5937.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 72 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2968//console This message is automatically generated. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466282#comment-13466282 ] Hadoop QA commented on HBASE-5937: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547122/HBASE-5937.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 72 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2969//console This message is automatically generated. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466288#comment-13466288 ] Ted Yu commented on HBASE-5937: --- @Flavio: Your recent rebase didn't work because HBASE-6476 was backed out. I think you don't need to frequently rebase. High-level comments would re-shape the patch dramatically. Thanks for your patience. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466307#comment-13466307 ] Ted Yu commented on HBASE-5937: --- It turns out the patch posted @ 29/Sep/12 10:14 applies cleanly to the current trunk. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466332#comment-13466332 ] Hadoop QA commented on HBASE-5937: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547128/HBASE-5937.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 72 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 157 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2971//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2971//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2971//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2971//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2971//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2971//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2971//console This message is automatically generated. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466394#comment-13466394 ] stack commented on HBASE-5937: -- Do javadoc in another patch, especially if not there originally. Its going to be hard enough landing this patch. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466401#comment-13466401 ] Ted Yu commented on HBASE-5937: --- bq. especially if not there originally The javadoc came from current HLog.java Looking at HLog interface, I wonder if interface Reader and interface Writer should be moved to their own files. That way we don't need to reference org.apache.hadoop.fs.FileSystem in HLog. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466402#comment-13466402
]
stack commented on HBASE-5937:
--
Looking at the patch, its looking great.
You fellas like how the new Interface looks or would you change it? Stuff like
isLowReplicationRollEnabled, getCoprocessorHost... these seems implementation
specific. And this mess:
{code}
public void hsync() throws IOException;
public void hflush() throws IOException;
public void sync() throws IOException;
public void sync(long txid) throws IOException;
{code}
... You are exposing scars HBase got following the ups and downs of HDFS
syncand the battle rages still.
I'd be fine w/ cleanup happening in another issue after this JIRA goes in.
appendNoSync should probably be append w/ a flag passed to append instead?
No problem doing mass rename of HLog to WAL after patch goes in in another
issue. Here is some review:
You have to do this cast?
{code}
+HBaseTestingUtility.setMaxRecoveryErrorCount(((FSHLog)
wal2).getOutputStream(), 1);
{code}
... more review to follow.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465599#comment-13465599 ] Hadoop QA commented on HBASE-5937: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546979/HBASE-5937.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 69 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2958//console This message is automatically generated. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466032#comment-13466032 ] Hadoop QA commented on HBASE-5937: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547074/HBASE-5937.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 72 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 157 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2962//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2962//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2962//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2962//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2962//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2962//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2962//console This message is automatically generated. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466067#comment-13466067
]
Ted Yu commented on HBASE-5937:
---
Thanks Flavio for the effort.
For HLogUtil.java and HLogFactory.java:
{code}
+ * Copyright 2010 The Apache Software Foundation
{code}
Year is not needed.
{code}
+public class HLogUtil {
{code}
Please add class javadoc and annotation for audience.
In HLog interface, I see methods without javadoc:
{code}
+public void registerWALActionsListener(final WALActionsListener listener);
+public boolean unregisterWALActionsListener(final WALActionsListener
listener);
{code}
Please add javadoc for each of the interface methods: they're the first place
where developer / user looks at when they want to understand HLog.
For a 260KB patch, review board would help reviewers a lot.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: HBASE-5937.patch, HBASE-5937.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460592#comment-13460592
]
Flavio Junqueira commented on HBASE-5937:
-
Thanks for responding, Stack.
bq. Sorry for not getting to your log. What have you been having to do to get
tests to pass? How did you fix TestMultiParallel? It is stuff to do w/ this
refactoring?
It was our bug.
bq. Currently Reader and Writer are Interfaces defined inside HLog. You get
one by calling a static method on HLog. You'd like to getReader non-static, an
invocation on a particular instance of HLog.
bq. That seems fine by me. It makes sense given what you are trying to do. It
is less flexible than what we currently have but its flexible because it
presumes a particular implementation of HLog.
It is simpler to leave getReader and getWriter as static methods. Given that a
reader/writer is for a concrete WAL, Ivan and I thought that it would be best
to have these methods available as instance methods. However, it is not looking
simple to implement because we don't have HLog objects available in all places
we need a reader or a writer, and the initialization of HLog objects makes it
tricky to instantiate it only to get a reader or a writer. At this point, I'm
tempted to leave them as static methods for now, unless anyone has a strong
preference otherwise.
bq. I hope you call your HLog Inteface WAL!
It is fine with me to make the change.
bq. I think this work trying to make an Interface for WAL is kinda important.
There is this bookeeping project but the multi-WAL dev – i.e. making the
regionserver write more than one WAL at a time (into HDFS) – could use the
result of this effort too.
BookKeeper provides the ability to write multiple concurrent logs, but if the
regionserver code is not prepared, then we won't be able to benefit from this
feature. Consequently, it does sound very important to have the regionserver
writing to more than one WAL at a time.
Currently there is one test failing consistently for me:
{noformat}
org.apache.hadoop.hbase.TestLocalHBaseCluster
{noformat}
and I believe the culprit is this:
{noformat}
WARNING! File system needs to be upgraded. You have version null and I want
version 7. Run the '${HBASE_HOME}/bin/hbase migrate' script.
2012-09-21 17:27:06,075 FATAL
[Master:0;perfectsalt-lm.barcelona.corp.yahoo.com,52906,1348241225714]
master.HMaster(1838): Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.util.FileSystemVersionException: File system needs to
be upgraded. You have version null and I want version 7. Run the
'${HBASE_HOME}/bin/hbase migrate' script.
{noformat}
Any clue of why this could be happening?
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460631#comment-13460631 ] stack commented on HBASE-5937: -- bq. Any clue of why this could be happening? Somehow the test is pointed at wrong fs? Did you mess w/ that? HBase, when it starts, it looks for a file named hbase.version. If present, it reads it to see that the version therein matches that of version the hbase software expects. We used this facility whenever on-fs formats changed in a way that required you to run a migration step before starting cluster. So, version == null makes me think hbase is looking in wrong place for the hbase.version file... looking in localfs rather than in hdf where it perhaps wrote it on startup? bq. ...and the initialization of HLog objects makes it tricky to instantiate it only to get a reader or a writer HLog construction is the way it is again because we presume one implementation only. I'd suggest you look at what it would take moving the heavyweight stuff done in HLog to an init or start method. NP having us change how we do the HLog setup in HBase. Perhaps it won't help much though as the Reader and Writer might want some of the heavy setup done? I'd also say that HLog is the way it is, not because it was designed, but because it evolved this way over the years. If you fellas want to startover, I'd say go for it: make a clean Interface that will work for our current hdfs use case and for the bkfs. We'll shoehorn it into a 0.98 or whatever suits your schedule. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13459755#comment-13459755
]
Flavio Junqueira commented on HBASE-5937:
-
I've been able to fix a few bugs, and I'm getting fewer failures/errors now:
{noformat}
Failed tests:
testExceptionFromCoprocessorDuringPut(org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort):
The put should have failed, as the coprocessor is buggy
Tests in error:
testZKClosingNodeVersionMismatch(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/740ffc6356202cd16e98dbe3ddf302cd
testCloseRegion(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/98327c03377cb4f0bf4a366bff09e68c
testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster): Master
not initialized after 200 seconds
{noformat}
I'll check those more closely. I was wondering of anyone has more thoughts on
the getReader/getWriter point I asked above. Thanks!
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460232#comment-13460232
]
stack commented on HBASE-5937:
--
[~fpj] Sorry for not getting to your log. What have you been having to do to
get tests to pass? How did you fix TestMultiParallel? It is stuff to do w/
this refactoring?
On your question{quote}I have also looked at making getReader/getWriter
part of HLog{quote}
What are you thinking? Currently Reader and Writer are Interfaces defined
inside HLog. You get one by calling a static method on HLog. You'd like to
getReader non-static, an invocation on a particular instance of HLog.
That seems fine by me. It makes sense given what you are trying to do. It is
less flexible than what we currently have but its flexible because it presumes
a particular implementation of HLog.
{quote}HLogInputFormat: Not clear how to instantiate HLog{quote}
This is a facility little used if ever. I'm surprised it not used more often.
Its a repair facility. You'd use it when you started a cluster somehow w/o
replaying WALs. You could use this class in a mapreduce job to quickly add the
edits from WAL back up into the cluster.
I took a look. What are you thinking constructors will look like for HLogs?
There'll be a factory? What will the factory take for arguments?
{quote}HLogPrettyPrinter: Executed through main calls in FSHLog and
HLogPrettyPrinter, so maybe we could pass necessary parameters{quote}
This is a tool for humans to look at contents of HLogs.
{quote}HLogSplitter: Have all parameters{quote}
This is the important one (smile)
{quote}HRegion: Have HLog object{quote}
Good... Its passed the HLog, right?
{quote}ReplicationSource: Not clear how to instantiate HLog{quote}
You know what this is about, right? This is how we do replication. We tail
the WALs and as the edits come in, we send them off to other clusters. We'll
need to be able to tail logs. Could we pass Replication an HLog instance?
I hope you call your HLog Inteface WAL!
{quote}I was also wondering if there are important side-effects in the case we
use the factory to get an HLog object just to get a reader or a writer{quote}
We'd have to change the current HLog constructor. It does a bunch of work when
created -- sets a sync'ing thread running (this syncing thread though is in
need of some cleanup), creates dirs and sets up first WAL. We wouldn't want it
doing this stuff if we wanted the instance just to do getReader/getWriter on it.
{quote}I have looked into the main constructor of FSHLog and I haven't been
able to convince myself that there is a way of executing it without throwing an
exception unnecessarily or having side-effects.{quote}
As it is currently written, yes.
I think this work trying to make an Interface for WAL is kinda important.
There is this bookeeping project but the multi-WAL dev -- i.e. making the
regionserver write more than one WAL at a time (into HDFS) -- could use the
result of this effort too.
Thanks Flavio.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451098#comment-13451098
]
Flavio Junqueira commented on HBASE-5937:
-
I've been looking at TestMultiParallel because and I'm getting this exception:
{noformat}
2012-09-07 19:38:31,560 WARN [Thread-371]
client.HConnectionManager$HConnectionImplementation(1020): Encountered problems
when prefetch META table:
org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for
table: multi_test_table, row=multi_test_table,\x00,99
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:166)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:56)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:135)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:132)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:394)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:132)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:107)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1017)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1071)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:959)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:916)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.submit(HConnectionManager.java:1917)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.doRetry(HConnectionManager.java:1957)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.processBatchCallback(HConnectionManager.java:2064)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.access$900(HConnectionManager.java:1859)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1848)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1827)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1003)
at
org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:246)
at
org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:213)
{noformat}
I'm not sure why the table is not present, and by skimming through the logs I
couldn't find any hint. If anyone has a hint, I'd be happy to hear, otherwise
I'll keep looking. I can also post the log if anyone is interested.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451103#comment-13451103 ] stack commented on HBASE-5937: -- Post log Flavio. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449640#comment-13449640
]
Flavio Junqueira commented on HBASE-5937:
-
I have moved static finals to HLog as suggested.
I have also looked at making getReader/getWriter part of HLog. I picked
getReader and extracted the classes using calling it. For each class, I tried
to determine if we can instantiate HLog or have already an instance of it. Here
is a summary:
# HLogInputFormat: Not clear how to instantiate HLog
# HLogPrettyPrinter: Executed through main calls in FSHLog and
HLogPrettyPrinter, so maybe we could pass necessary parameters
# HLogSplitter: Have all parameters
# HRegion: Have HLog object
# ReplicationSource: Not clear how to instantiate HLog
I was also wondering if there are important side-effects in the case we use the
factory to get an HLog object just to get a reader or a writer. I have looked
into the main constructor of FSHLog and I haven't been able to convince myself
that there is a way of executing it without throwing an exception unnecessarily
or having side-effects.
The errors and failures I'm getting for tests now are:
{noformat}
Failed tests:
testRetrying(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor): reader:
count=2, t=null
testExceptionFromCoprocessorDuringPut(org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort):
The put should have failed, as the coprocessor is buggy
testWALCoprocessorReplay(org.apache.hadoop.hbase.coprocessor.TestWALObserver)
test2727(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay)
testReplayEditsWrittenViaHRegion(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay)
testReplayEditsAfterPartialFlush(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
expected:30 but was:20
testReplayEditsWrittenIntoWAL(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
Flushcount=0
test2727(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed)
testReplayEditsWrittenViaHRegion(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed)
testReplayEditsAfterPartialFlush(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
expected:30 but was:20
testReplayEditsWrittenIntoWAL(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
Flushcount=0
{noformat}
{noformat}
testZKClosingNodeVersionMismatch(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/67f6e4f3629fa0bb34fbd5cb604ff44e
testCloseRegion(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/e2b47297b32db6a95e2a310be053172f
testDataCorrectnessReplayingRecoveredEdits(org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster):
Failed 1 action: ExecutionException: 1 time,
testReplayEditsAfterRegionMovedWithMultiCF(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
Failed after attempts=10, exceptions:(..)
testReplayEditsAfterRegionMovedWithMultiCF(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
Failed after attempts=10, exceptions:(..)
{noformat}
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13438734#comment-13438734
]
Ivan Kelly commented on HBASE-5937:
---
{quote}
Can you factory it in places like HLogInputFormat?
{quote}
HLogInputFormat among a few funny ones which I wasn't sure how to deal with
(one of the others is the PrettyPrinter). The problem with these is that they
take a individual log file, rather than a whole log instance. This makes it
more complicated to use it with HLogFactory, as createHLog asks for the whole
log directory. We could get the log directory by using Path.getParent(), but
that seemed messy to me at the time. Otherwise, yes, the solution to
HLogUtil.createReader - HLog#createReader is to instantiate a HLog where
needed. We just haven't gotten that far yet.
{quote}
Whats FSLog? An HDFSLog?
{quote}
Exactly. FileSystem doesn't necessarily have to be HDFS so FSLog is a better
name.
{quote}
Can you not get HLogFactory.createHLog into the places where we have getReader
now – e.g. in HRegion (Should HRegion even be concerned w/ HLog/WAL? Only
RegionServer should be?)?
{quote}
As I understand it now, HRegion only creates the HLog for the META and ROOT
tables, which are managed from the Master and as such, do not have access to a
RegionServer. Perhaps this could be refactored a little to make HRegion always
receive a preconstructed HLog.
{quote}
Should HLog Interface be instead named WAL?
Is it right that the HLog Interface takes an fs? That OK for you lads? You'll
be doing a bookkeeper FS?
HLog Interface seems fat. We need all those methods?
{quote}
The current patch is a first cut to give you guys an idea of where we are, but
it's quite far from what we imagine the final interface from looking like. At
the moment, what we want to do is refactor all the HLog code so that everything
that access HLog is going through well defined interfaces. Once that is done,
we can look at the interfaces to see where the implementation specific stuff
(like Path, Filesystem etc) is leaking out, and work to resolve them.
I'll try and find time to get back to do some coding on this (i.e. fix tests,
refactor createReader) this week.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13438842#comment-13438842 ] stack commented on HBASE-5937: -- bq. The problem with these is that they take a individual log file, rather than a whole log instance. Can we change this boss and pass in an HLog or whatever instance? bq. We just haven't gotten that far yet. np bq. Perhaps this could be refactored a little to make HRegion always receive a preconstructed HLog. Yes. The fact that HRegion is making HLog instances smells. And +1 on your approach. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437158#comment-13437158
]
stack commented on HBASE-5937:
--
bq. In the case we make them part of the interface, we will need to find a way
of making an HLog object available in those places.
If these are HLog reader and writers, yeah, should be in the HLog interface I'd
say rather than in an HLogUtil. Can you factory it in places like
HLogInputFormat?
Why move HLog defines into HLogUtil? They seem like core defines rather than
util defines: e.g. HLog.SPLITTING_EXT.
Yeah, this seems like a basic one that needs solving:
{code}
- reader = HLog.getReader(this.fs, edits, conf);
+ reader = HLogUtil.getReader(this.fs, edits, conf);
{code}
You'll need a 'wal' object, one that has been appropriately created dependent
on what WAL engine has been put in place.
Can you not get HLogFactory.createHLog into the places where we have getReader
now -- e.g. in HRegion (Should HRegion even be concerned w/ HLog/WAL? Only
RegionServer should be?)?
Whats FSLog? An HDFSLog?
Should HLog Interface be instead named WAL?
Is it right that the HLog Interface takes an fs? That OK for you lads? You'll
be doing a bookkeeper FS?
HLog Interface seems fat. We need all those methods?
Seems good so far?
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Li Pi
Priority: Minor
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434195#comment-13434195 ] Flavio Junqueira commented on HBASE-5937: - I wanted to report the progress Ivan and I have made on this issue so far. We have a repository that we have been working on, just in case anyone is interested in giving comments on our changes so far: https://github.com/fpj/hbase Here are some issues we have come across and we have in our todo list: # Failing tests: There are a number of tests failing and we are still investigating the causes; # getReader/createWriter: We moved these methods to HLogUtil, but we can't agree on whether they should be part of the new HLog interface or not. The main issue is that getReader is called from a number of places where we have no HLog object available. In the case we make them part of the interface, we will need to find a way of making an HLog object available in those places. # HLogSplitter: It should be part of HLogFactory. # Protected methods of FSHLog: There are a number of calls in the tests to protected methods of FSHLog. We solved it so far by type-casting, but it doesn't look very clean and I'm not very happy about that. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434231#comment-13434231 ] Zhihong Ted Yu commented on HBASE-5937: --- @Li: Can I assign this issue to Flavio ? Thanks Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434247#comment-13434247
]
Zhihong Ted Yu commented on HBASE-5937:
---
Thanks Flavio and Ivan for tackling this issue.
w.r.t. reader / writer creation, we can leave them in HLogUtil for now. They
don't belong to FSHLog or HLogFactory.
For HLog interface, please add javadoc for each of its methods.
{code}
public interface Reader {
void init(FileSystem fs, Path path, Configuration c) throws IOException;
{code}
If HLog.Reader is FileSystem backed, I would assume that HLog is backed by
FileSystem as well.
Can we declare the first parameter as Object ?
FSHLog can declare FSReader which extends HLog.Reader that casts first
parameter from init() as FileSystem.
Please tell us which tests fail so that other people can help.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Li Pi
Priority: Minor
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434504#comment-13434504
]
Zhihong Ted Yu commented on HBASE-5937:
---
Here is partial list of failed tests:
{code}
test2727(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed)
testReplayEditsWrittenViaHRegion(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed)
testReplayEditsAfterPartialFlush(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
expected:30 but was:20
testReplayEditsWrittenIntoWAL(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
Flushcount=0
test2727(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay)
testReplayEditsWrittenViaHRegion(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay)
testReplayEditsAfterPartialFlush(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
expected:30 but was:20
testReplayEditsWrittenIntoWAL(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
Flushcount=0
testCorrectnessWhenMasterFailOver(org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing)
...
Tests in error:
queueFailover(org.apache.hadoop.hbase.replication.TestReplication): No server
address listed in .META. for region
test,mmm,1344966925833.ca73ab045b69ab49cec353012f01fd35. containing row mmm
testDrainingServerWithAbort(org.apache.hadoop.hbase.TestDrainingServer): test
timed out after 3 milliseconds
testFlushCommitsWithAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
testBatchWithPut(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
testBatchWithDelete(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
testHTableDeleteWithList(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
testBatchWithManyColsInOneRowGetAndPut(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
testBatchWithIncrementAndAppend(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
testBatchWithMixedActions(org.apache.hadoop.hbase.client.TestMultiParallel):
multi_test_table
test2772(org.apache.hadoop.hbase.client.TestScannerTimeout): Failed after
attempts=10, exceptions:(..)
test3686a(org.apache.hadoop.hbase.client.TestScannerTimeout): t
test3686b(org.apache.hadoop.hbase.client.TestScannerTimeout): t
testZKClosingNodeVersionMismatch(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/7f81d00867e4832617a414d6d970c8ab
testCloseRegion(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/296e7298f90ab0bec536aba4b039fa84
{code}
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Li Pi
Priority: Minor
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428181#comment-13428181 ] Flavio Junqueira commented on HBASE-5937: - Just to make sure, there is nothing done related this jira yet, yes? Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428184#comment-13428184 ] Zhihong Ted Yu commented on HBASE-5937: --- Right. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268056#comment-13268056 ] Zhihong Yu commented on HBASE-5937: --- We should define clearer goals for the interface. Is this mostly for multi-WAL support ? Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268074#comment-13268074 ] Li Pi commented on HBASE-5937: -- Mostly. But you could use it for other things too. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
