[
https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714728#action_12714728
]
Luca Telloli commented on HADOOP-5188:
--------------------------------------
Konstantin, to follow up on Flavio's comment, let me use BookKeeper as an
example.
To use bookkeeper as a log device I need to open a client. If I open it in
EditLogOutputStream that client should be destroyed upon a close() call and
would be recreated each time a new EditLogOutputStream is called: I don't think
this is the right way to proceed. LogDevice would create an instance of such a
client and keep it persistent thourough the life of a server process (namenode,
etc). This motivates the insufficiency of the current EditStreams.
So far HDFS has been working only with files for logging. BackupNode is a
special type of process that, although it registers a stream, at the end uses
files again. I don't think that every other log device should spawn a new
process similar to BackupNode for logging, although I assume that in production
users will choose only one device.
Finally, it would be hard to integrate Bookkeeper with the current streams
because BookKeeper ledgers do not support append. Files do, bookkeeper doesn't
(instead of appending, we create another "ledger" - the bookkeeper log file).
It's part of the diversity of the two "devices", it's another argument in favor
of the need of an abstraction.
LogDevice has been passing so far all unit tests successfully and it guarantees
compatibility with the current implementation, so it seems a safe absstraction.
I believe it's also a good one but I'd be delighted to hear a concrete proposal
to make it better or replace it.
Other points:
1. HADOOP-5188 does introduce LogDevice and I agree it's not fully uses yet
thorough the code. But if we want to keep it, we'll do the modifications
eventually.
2. Method EditLogFileOutputStream.switchEditStreams() should eventually contain
part of the rollEditLog() code. In the case of the BackupNode you probably know
better what should be there, and if you wanna contribute to it, you're very
welcome.
3. The use of instanceof has been reduced and it will disappear in my next
version, since I introduced a getType() method. Reflection is used at the end
only to instantiate the correct log device implementation
> Modifications to enable multiple types of logging
> --------------------------------------------------
>
> Key: HADOOP-5188
> URL: https://issues.apache.org/jira/browse/HADOOP-5188
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Luca Telloli
> Fix For: 0.21.0
>
> Attachments: HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch,
> HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.pdf
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
