Hi David!

A FileSystem class is an abstraction for the file system. It doesn't make
sense to do an hsync on a file system (should the file system sync all
files currently open / just the user's etc.) . With appropriate flags maybe
you can make it make sense, but we don't have that functionality.

When you create() a file in the file system, you get back a
FSDataOutputStream on which you can call the hsync() method . Doesn't that
make more sense? On that method call, the buffer from the client is flushed
to the pipeline, and an RPC goes to the NameNode to update the length
<https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L3146>
(which is a fair amount of work). This is obviously not a scalable
solution. Perhaps you might want to look at Kafka? If you don't need it to
scale, and are fine with hammering the NameNode (really you shouldn't be),
then maybe HDFS inotify
<https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java>
can help?

HTH,
Ravi

On Wed, Aug 9, 2017 at 5:37 AM, David Robison <david.robi...@psgglobal.net>
wrote:

> I understand that, when writing to a file, I can force it to update its
> length on the namenode by using the following command:
>
>
>
> ((DFSOutputStream) imageWriter.getWrappedStream()
> ).hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH));
>
>
>
> Is there a way to force the update without having to open a
> DFSOutputStream? Can I do this from the FileSystem class or some other Java
> class? The reason for this is that I am mostly writing to HDFS and only
> occasionally reading. However, when I go to read, I am most often reading
> the most recent data written (reading the end of the file not the
> beginning). If I could force the length update at the time of reading that
> would save time by not having to make sure I update the length every time I
> write to the file (which is about once per second).
>
>
>
> Thanks, David
>
>
>
> *David R Robison*
>
> *Senior Systems Engineer*
>
> O. +1 512 247 3700 <(512)%20247-3700>
>
> M. +1 757 286 0022 <(757)%20286-0022>
>
> david.robi...@psgglobal.net
>
> *www.psgglobal.net <http://www.psgglobal.net/>*
>
> [image: cid:image003.png@01D19182.F24CA3E0]
>
> *Prometheus Security Group Global, Inc.*
>
> 3019 Alvin Devane Boulevard
>
> Building 4, Suite 450
>
> Austin, TX 78741
>
> [image: cid:image003.png@01D19182.F24CA3E0]
>
>
>

Reply via email to