[
https://issues.apache.org/jira/browse/HDFS-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634063#comment-16634063
]
Wei-Chiu Chuang edited comment on HDFS-13877 at 10/1/18 2:04 PM:
-----------------------------------------------------------------
Thanks [~smeng] for the patch. I should have suggested you to implement
DistributedFileSystem#snapshotDiffReportListingRemoteIterator.
Quoting my comments in HDFS-13052:
{quote}I'm late to review this (coming from HDFS-13877). While this Jira
provides a handy snapshotdiff api, in practice this is not usable in production.
See: HDFS-12594 and HDFS-12165. In extreme cases I've seen
getSnapshotDiffReport RPC sending 2GB protobuf message and failed. Even in
not-so-extreme cases, since webhdfs server side runs in NameNode process, NN
heap usage change like that can easily fail SLA or even lead to fail over.
Instead, we should implement something equivalent to
DistributedFileSystem#snapshotDiffReportListingRemoteIterator. Use the
interface implemented in HDFS-12594 and return an iterator.
{quote}
Note that httpfs server runs as an independent process, typically not sharing
the same host as NN, so the performance impact is less.
was (Author: jojochuang):
Thanks [~smeng] for the patch. I should have suggested you to implement
DistributedFileSystem#snapshotDiffReportListingRemoteIterator.
Quoting my comments in HDFS-13052:
{quote}I'm late to review this (coming from HDFS-13877). While this Jira
provides a handy snapshotdiff api, in practice this is not usable in production.
See: HDFS-12594 and HDFS-12165. In extreme cases I've seen
getSnapshotDiffReport RPC sending 2GB protobuf message and failed. Even in
not-so-extreme cases, since webhdfs server side runs in NameNode process, NN
heap usage change like that can easily fail SLA or even lead to fail over.
Instead, we should implement something equivalent to
DistributedFileSystem#snapshotDiffReportListingRemoteIterator. Use the
interface implemented in HDFS-12594 and return an iterator.
{quote}
> HttpFS: Implement GETSNAPSHOTDIFF
> ---------------------------------
>
> Key: HDFS-13877
> URL: https://issues.apache.org/jira/browse/HDFS-13877
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: httpfs
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDFS-13877.001.patch, HDFS-13877.001.patch
>
>
> Implement GETSNAPSHOTDIFF (from HDFS-13052) in HttpFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
