IMO, it's okay to keep the files in HDFS, at startup you can load them in memory which will be one time activity in application lifetime. Considering they are format/config files, it won't be difficult to cache it in memory of your current operator (no need to caching operator separately).
-Priyanka On Wed, Aug 10, 2016 at 7:06 PM, Mukkamula, Suryavamshivardhan (CWM-NR) < [email protected]> wrote: > Hello Team, > > I have a use case where I am trying to read number of input files with > different formats(may be 100 source systems) and I have to parse each > record. > > To achieve this we need to have metadata(configuration files) of the input > file structures. Can you please let me know that, having this metadata on > HDFS is good or having the metadata in some sql/oracle database is good? > > Assume that I don’t have caching operator for now, I need to get the > configuration file for each source and I have to parse them at run time. To > do this going to HDFS is good or going to database is good? > > Regards, > Surya Vamshi > > > _______________________________________________________________________ > > If you received this email in error, please advise the sender (by return > email or otherwise) immediately. You have consented to receive the attached > electronically at the above-noted email address; please retain a copy of > this confirmation for future reference. > > Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur > immédiatement, par retour de courriel ou par un autre moyen. Vous avez > accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à > l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de > cette confirmation pour les fins de reference future. > >
