[
https://issues.apache.org/jira/browse/NIFI-1924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305916#comment-15305916
]
Andre commented on NIFI-1924:
-----------------------------
[~jeremy.dyer]
did my first tests this evening and got no communication to the webhdfs
endpoint + some errors:
{code}
<configuration>
<property>
<name>fs.defaultFS</name>
<value>webhdfs://hostname:14000/webhdfs/v1/</value>
</property>
<property>
<name>fs.webhdfs.impl</name>
<value>org.apache.hadoop.hdfs.web.WebHdfsFileSystem</value>
</property>
</configuration>
{code}
Note that I am trying to call a httpfs instead of webhdfs but this doesn't seem
to be the issue as no TCP traffic is seen.
{code}
o.apache.nifi.processors.hadoop.ListHDFS
java.io.FileNotFoundException: File /user/test does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem
.java:376) ~[hadoop-common-2.6.2.jar:na]
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1485) ~[ha
doop-common-2.6.2.jar:na]
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1525) ~[ha
doop-common-2.6.2.jar:na]
at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem
.java:570) ~[hadoop-common-2.6.2.jar:na]
at
org.apache.nifi.processors.hadoop.ListHDFS.getStatuses(ListHDFS.java:356)
~[nifi-hdfs-processors-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
{code}
Note I am testing against 0.4.x but this piece of the code hasn't gone through
much change lately.
> Create set of WebHDFS processors
> --------------------------------
>
> Key: NIFI-1924
> URL: https://issues.apache.org/jira/browse/NIFI-1924
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Andre
>
> Create processors to handle,
> Supported commands for WebHDFS V1
> * GET -> GetWebHDFS
> * OPEN
> * GETFILESTATUS
> * LISTSTATUS
> * GETCONTENTSUMMARy
> * GETFILECHECKSUM
> * GETHOMEDIRECTORY
> * GETDELEGATIONTOKEN
> * GETDELEGATIONTOKENS
> * GETXATTRS
> * LISTXATTRS
> * CHECKACCESS
> * PUT -> PutWebHDFS
> * CREATE
> * MKDIRS
> * CREATESYMLINK
> * RENAME
> * SETREPLICATION
> * SETOWNER
> * SETPERMISSION
> * SETTIMES
> * RENEWDELEGATIONTOKEN
> * CANCELDELEGATIONTOKEN
> * CREATESNAPSHOT
> * RENAMESNAPSHOT
> * SETXATTR
> * REMOVEXATTR
> * POST -> PostWebHDFS
> * APPEND
> * CONCAT
> * TRUNCATE
> * DELETE -> DeleteWebHDFS
> * DELETE
> * DELETESNAPSHOT
> operations against WebHDFS as discussed here:
> http://mail-archives.apache.org/mod_mbox/nifi-users/201604.mbox/%3CCABtjSZk2KLJJVk%2Byk-ySioPxVUV5SLHcN%2BaWMZs1ARpkfy%2B0Ow%40mail.gmail.com%3E
> Required Processor Properties:
> * Namenode Host
> * WebHDFS port
> * WebHDFS API version. Default to v1. Dropdown list of available options.
> Currently there is only v1 but would be good to have that in place to show
> our intentions.
> * Hadoop configuration files. core-site.xml and hdfs-site.xml. These
> configurations files should not be required to keep things lite. However they
> would be required to access a kerberized Hadoop cluster. If these
> configuration files are present they should take precedence over the
> properties set in NameNode host and WebHDFS port.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
