Thanks for response, one more question. can we append new data to the existing ORC file. using ORC writer i think we cant append new Data once the writer is closed, but can we do it if i use Hive?
On Wed, Jul 27, 2016 at 4:26 PM, Owen O'Malley <[email protected]> wrote: > You can use org.apache.orc.Writer.writeIntermediateFooter(). That will > write a temporary footer to the file and flush it to hdfs. > writeIntermediateFooter will return the length of the file after the flush, > which should be passed in to OrcFile.ReaderOptions.maxLength(). In Hive, > we write the lengths in a side file so that if the writer dies suddenly, it > can always get the last footer. Look at OrcAcidUtils.getLastFlushLength(). > > .. Owen > > On Wed, Jul 27, 2016 at 11:27 AM, praveen reddy < > [email protected]> wrote: > >> Hi, >> >> i am creating orc file based on particular action(kind of action user >> does from UI) and that action may happen in future as well. so ,one ORC >> file will be created for particular action and data will keep appending to >> that file when that action happens again in future. >> >> is there a way i can flush the data to HDFS without closing the Writer >> object. i will close the Writer object when the file on HDFS reaches >> particular length. only when i create a new file, writer on old file should >> be closed. >> >> currently i can see only close , flushes the data to HDFS. i don't want >> to close the file, keep writing to the file, flush it and close write only >> when file is of particular size. >> >> is there a way i can achieve it? >> >> Thanks, >> Praveen >> > >
