You can use org.apache.orc.Writer.writeIntermediateFooter(). That will write a temporary footer to the file and flush it to hdfs. writeIntermediateFooter will return the length of the file after the flush, which should be passed in to OrcFile.ReaderOptions.maxLength(). In Hive, we write the lengths in a side file so that if the writer dies suddenly, it can always get the last footer. Look at OrcAcidUtils.getLastFlushLength().
.. Owen On Wed, Jul 27, 2016 at 11:27 AM, praveen reddy < [email protected]> wrote: > Hi, > > i am creating orc file based on particular action(kind of action user does > from UI) and that action may happen in future as well. so ,one ORC file > will be created for particular action and data will keep appending to that > file when that action happens again in future. > > is there a way i can flush the data to HDFS without closing the Writer > object. i will close the Writer object when the file on HDFS reaches > particular length. only when i create a new file, writer on old file should > be closed. > > currently i can see only close , flushes the data to HDFS. i don't want to > close the file, keep writing to the file, flush it and close write only > when file is of particular size. > > is there a way i can achieve it? > > Thanks, > Praveen >
