See in 0.20 hbase branch, in HLog constructor, where we figure if
syncFs is available:
// Test if syncfs is available.
this.append = isAppend(conf);
Method m = null;
if (this.append) {
try {
m = this.writer.getClass().getMethod("syncFs", new Class<?> []{});
LOG.debug("Using syncFs--hadoop-4379");
} catch (SecurityException e) {
throw new IOException("Failed test for syncfs", e);
} catch (NoSuchMethodException e) {
// This can happen
LOG.info("syncFs--hadoop-4379 not available" );
}
}
this.syncfs = m;
Then, how HLog#sync will call syncFs if available:
public void sync() throws IOException {
lastLogFlushTime = System.currentTimeMillis();
if (this.append && syncfs != null) {
try {
this.syncfs.invoke(this.writer, NO_ARGS);
} catch (Exception e) {
throw new IOException("Reflection", e);
}
} else {
this.writer.sync();
}
this.unflushedEntries.set(0);
syncTime += System.currentTimeMillis() - lastLogFlushTime;
syncOps++;
}
Then, on split of logs, see the recoverLog method for the bit where we
wait around till the open for append works.
Ask more if still not clear,
St.Ack
On Tue, Jan 26, 2010 at 3:50 PM, Nicolas Spiegelberg
<nspiegelb...@facebook.com> wrote:
> Sounds like I also have 0.20.3RC merging as well. Version mistake on my
> behalf. We have a 0.20.3-dev that must've missed the 'syncfs' changes. I
> was merging based solely on the 0.21 trunk. I'll look into the 0.20.3 code
> some more.
>
> I have done the HDFS-200 merge & the trunk group commit, now I need to
> reconcile that with the 0.20.3RC code since we don't currently plan on
> merging HDFS-265.
>
> Thanks!
> Nicolas Spiegelberg
>
> On 1/26/10 2:02 PM, "Stack" <st...@duboce.net> wrote:
>
>> HBase 0.20 had a hack that would recognize the presence of Dhruba's
>> HDFS-200. If it had been applied, then we'd do the open-for-append,
>> close, and reopen to recover edits written to an unclosed WAL/HLog
>> file (Grep 'syncfs' in HLog on the 0.20 branch).
>>
>> In HBase TRUNK, the above hackery was stripped out. In TRUNK we are
>> leaning on the new hflush/HDFS-265 rather than HDFS-200. For hflush,
>> when we do FSDataInputStream::available(), its returning the 'right'
>> answer (WALReaderFsDataInputStream::getPos() was added before an API
>> was available. HBASE-2069 is about using the new API instead of this
>> getPos fancy-dancing).
>>
>> It sounds like you need to do a bit of merging of TRUNK group commit
>> and the old hbase code that exploited HDFS-200?
>>
>> St.Ack
>>
>> On Tue, Jan 26, 2010 at 12:35 PM, Nicolas Spiegelberg
>> <nspiegelb...@facebook.com> wrote:
>>> Hi,
>>>
>>> I am trying to backport the HLog group commit functionality to Hbase 0.20.
>>> For proper reliability, I am working with Dhruba to get the 0.21 syncFs()
>>> changes from HDFS ported back to HDFS 0.20 as well. When going through a
>>> peer review of the modified code, my group had a question about the
>>> SequenceFileLogReader.java (WALReader). I am hoping that you guys could be
>>> of assistance.
>>>
>>> I know that there is an open issue [HBASE-2069] where Hlog::splitLog() does
>>> not call DFSDataInputStream::getVisibleLength(), which would properly sync
>>> hflushed, but unclosed, file lengths. I believe the current workaround is
>>> to
>>> open an HDFS file in append mode & then close, which would cause the
>>> namenode
>>> to get updates from the datanodes. However, I don¹t see that shim present
>>> in
>>> Hlog::splitLog() on the 0.21 trunk. Is this a pending issue to fix or is
>>> calling FSDataInputStream::available() within
>>> WALReaderFsDataInputStream::getPos() sufficient to force the namenode to
>>> sync
>>> up with the datanodes?
>>>
>>> Nicolas Spiegelberg
>>>
>
>
