What about using flume to consume the files and send to HDFS? It depends
upon what security implication you are looking for.
It will be easier if you can look into answer of similar question below.
Is Jboss server process is pushing file to HDFS or some other process is
pushing file to HDFS?
The intermediate server, is doing some security check on data or user
access? What processing is done on intermediate server? if not how it is
different from sending the file directly to HDFS?
Regards,
Abhishek
On Sat, May 12, 2012 at 3:56 AM, samir das mohapatra <
samir.help...@gmail.com> wrote:
> Hi financeturd <financet...@yahoo.com>,
> My Point of view second step like bellow is the good approach
>
> {Separate server} <-- {JBoss server}
> and then
> {Separate server} --> HDFS
>
> thanks
> samir
>
> On Sat, May 12, 2012 at 6:00 AM, financeturd financeturd <
> financet...@yahoo.com> wrote:
>
> > Hello,
> >
> > We have a large number of
> > custom-generated files (not just web logs) that we need to move from our
> > JBoss servers to HDFS. Our first implementation ran a cron job every 5
> > minutes to move our files from the "output" directory to HDFS.
> >
> > Is this recommended? We are being told by our IT team that our JBoss
> > servers should not have access to HDFS for security reasons. The files
> > must be "sucked" to HDFS by other servers that do not accept traffic
> > from the outside. In essence, they are asking for a layer of
> > indirection. Instead of:
> > {JBoss server} --> {HDFS}
> > it's being requested that it look like:
> > {Separate server} <-- {JBoss server}
> > and then
> > {Separate server} --> HDFS
> >
> >
> > While I understand in principle what is being said, the security of
> having
> > processes on JBoss servers writing files to HDFS doesn't seem any worse
> > than having Tomcat servers access a central database, which they do.
> >
> > Can anyone comment on what a recommended approach would be? Should our
> > JBoss servers push their data to HDFS or should the data be pulled by
> > another server and then placed into HDFS?
> >
> > Thank you!
> > FT
>
